This document is created with the unregistered version of CHM2PDF Pilot

Copyright © 2001 by Kalen Delaney

This document is created with the unregistered version of CHM2PDF Pilot

34

PUBLISHED BY
Microsoft Press
A Division of Microsoft Corporation
One Microsoft Way
Redmond, Washington 98052-6399

Copyright © 2001 by Kalen Delaney

All rights reserved. No part of the contents of this book may be reproduced or transmitted in any form
or by any means without the written permission of the publisher.

Library of Congress Cataloging-in-Publication Data

Delaney, Kalen.
Inside Microsoft SQL Server 2000 / Kalen Delaney.
p. cm.
Includes bibliographical references and index.
ISBN 0-7356-0998-5
1. Client/server computing. 2. SQL server. 3. Relational databases. I. Title.

QA76.9.C55 D43 2000

005.75'85--dc21 00-059280

Printed and bound in the United States of America.

1 2 3 4 5 6 7 8 9 QWT 6 5 4 3 2 1

Distributed in Canada by Penguin Books Canada Limited.

A CIP catalogue record for this book is available from the British Library.

Microsoft Press books are available through booksellers and distributors worldwide. For further information
about international editions, contact your local Microsoft Corporation office or contact Microsoft Press
International directly at fax (425) 936-7329. Visit our Web site at mspress.microsoft.com. Send comments to
[email protected].

ActiveX, BackOffice, FoxPro, JScript, Microsoft, Microsoft Press, MS-DOS, MSDN, MSN, Visual
Basic, Visual C++, Visual FoxPro, Visual InterDev, Visual J++, Visual Studio, Win32, Windows, and
Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the United States
and/or other countries. Other product and company names mentioned herein may be the trademarks of
their respective owners.

Unless otherwise noted, the example companies, organizations, products, people, and events depicted herein
are fictitious. No association with any real company, organization, product, person, or event is intended or
should be inferred.

Acquisitions Editor: David Clark

Project Editor: Rebecca McKay
Manuscript Editor: Ina Chang
Technical Editors: Marc Young and Jean Ross
This document is created with the unregistered version of CHM2PDF Pilot

34
For Dan,
forever.
This document is created with the unregistered version of CHM2PDF Pilot

34

Foreword
This is the definitive companion to Microsoft SQL Server 2000. The authors have combined the perspectives of the
developer and the user into a readable and comprehensive guide. Ron Soukup led the SQL Server development team
for a decade, and he continues to be one of its key architects. Kalen Delaney has been working with the application
designers and developers for almost that long. This is their guide to why SQL Server is the way it is, and how it
should be used. The guide complements the reference material in SQL Server Books Online.

The book begins with the inside story of how the PC database giant Ashton-Tate partnered with Microsoft and a
startup (Sybase) to bring SQL to the OS/2 marketplace. The book then traces the amazing history of the product and
the partners. This first-hand account makes this a must-read. I cannot think of a more amazing and entertaining story
in our industry.

The book then takes the reader on a whirlwind tour of SQL Server, describing the key features and pointing out
some highlights. It also goes "beneath the covers" and describes how SQL Server works inside. First-time readers
might find this quite detailed, but it lays the groundwork for much that follows. Subsequent chapters explain how to
design for SQL Server—giving sage advice on application design, database design, physical data layout, and
performance tuning. This advice is based on watching customers use (and misuse) the product over many years. SQL
Server and the Microsoft Windows environment are unique in many ways. These chapters cover the standard design
issues but focus primarily on the things that are unique to SQL Server. Incidental to this, Ron and Kalen show many
of the pitfalls and common errors that designers and programmers make. They also convey their sense of good and
bad design. Anyone planning to do an implementation using SQL Server would do well to first read this book.

The book covers virtually all the extensions that SQL Server has made to the standard SQL language. Ron and
Kalen take pride in describing these features. They often explain why features were introduced, how they compare to
the competition, and how they work. This book is not an SQL tutorial, but it does an excellent job of covering the
intermediate and advanced features of SQL and SQL Server. The descriptions are accompanied by detailed
examples that are all on the companion CD.

The book also explains how to install, administer, and tune SQL Server. These chapters contain essential information
that I have never seen explained elsewhere. It is very easy to get started with SQL Server—perhaps too easy. Some
customers just install it and start using it without thinking much. Ron and Kalen walk the designer through (1) capacity
planning, (2) hardware acquisition, (3) Windows, network, and RAID configuration, (4) SQL installation and
licensing, (5) security policies, (6) operations procedures, (7) performance measurement, and (8) performance tuning.
The book provides a valuable checklist for anyone planning to setup or operate an SQL Server system.

The book devotes several chapters to understanding performance, concurrency, and recovery issues. Throughout the
book, there is an emphasis on designing for client-server and Internet environments. In these environments, the server
must process business-rules (stored procedures) or set-oriented requests, rather than record-at-a-time requests. The
book explains both traditional TransactSQL stored procedures, as well as user-defined external procedures and OLE
Automation procedures.

The book gives a clear picture of how SQL Server transactions work. It first presents tutorial material on the ACID
properties, isolation levels, and locking. It then explains how SQL server does locking, logging, checkpointing and
recovery. It has a full chapter on how backup/restore work and how to protect your database from failures. It also
explains how SQL Server uses the Windows Cluster feature for high-availability. This presentation is exceptionally
clear. Understanding these issues is key to designing high-traffic and high-availability SQL Servers.

SQL Server database replication is described in depth—especially the features that are new in SQL Server 2000. It
also describes the many new features in SQL Server 2000 like table-valued functions, variants, full-text indexing, data
This document is created with the unregistered version of CHM2PDF Pilot

mining, data analysis, and multi-instance support.

In summary, this is the essential companion to SQL Server. It is an invaluable reference for the administrator and the
designer—I look something up in it nearly every week. It is a first-hand account of how SQL Server works, why it
works that way, and how to use it. We are all glad Ron and Kalen took the time to write this invaluable book. I
certainly learned a lot from it. I guess you can too.

Jim Gray

Senior Researcher

Microsoft San Francisco Research Lab

This document is created with the unregistered version of CHM2PDF Pilot

34

Preface
The most wonderful thing about writing a book and having it published is getting feedback from you, the readers. Of
course, it's also the worst thing. I wanted the previous edition, Inside SQL Server 7.0, to satisfy everybody, as
impossible as that might sound. When I got e-mail from readers thanking me for the book, I would feel that all the
time I put into it was well spent. When I got e-mail telling me that the book was not satisfactory, I would worry and
fret and try to figure out how to make it better the next time. In fact, most of the negative comments were from people
who complained about what wasn't included in the book, as if they were seeing the cup half empty instead of half full.
As I said in my preface to the last edition, this book doesn't cover the entire product. The focus, as the name Inside
implies, is on the core engine, in particular the query processor and the storage engine. The book doesn't talk about
client programming interfaces, heterogeneous queries, or replication. In addition, some internal operations aren't
discussed, such as security. However, unlike in the previous edition, I do include a discussion of the backup and
restore operations. But I had to draw the line somewhere so the book wouldn't be twice as long and so I wouldn't still
be writing it and avoiding calls from my editors asking when the next chapter would arrive.

For Inside SQL Server 7.0, I basically followed Ron Soukup's original outline. I made changes as required for the
new version of the product, some of them quite substantial, but the organization was basically as it had been in the
very first edition. In Inside SQL Server 2000, I've consolidated most of the information about indexes into a separate
chapter. The previous editions discussed transactions, stored procedures, and triggers all in one chapter; with the
addition of user-defined functions and new trigger capabilities in SQL Server 2000, I've split the discussion into two
chapters. In the previous editions, query processing and tuning were combined into one huge chapter. I separated
these topics into two chapters, one dealing with the internals of query processing and how the SQL Server optimizer
works and another chapter that provides suggestions for using what you've learned to write better performing queries.
Finally, I combined two chapters dealing with Transact-SQL programming into one. When Ron's first edition came
out in 1997, there were no good books on the market about complex programming using the Microsoft dialect of the
SQL language. That is no longer true, and suggested reading list at the end of this book mentions three excellent
books about writing very advanced queries using Transact-SQL. I am grateful to the authors of those books for
allowing me to minimize my discussion of programming and to concentrate more on the workings of the SQL Server
engine.

I hope you find value here even if I haven't covered every single SQL Server topic that you're interested in. You can
let me know what you'd like to learn more about, and I can perhaps refer you to other books or whitepapers. Or
maybe I'll write an article for TechNet or SQL Server Magazine. You can contact me via my Web site at
www.InsideSQLServer.com.

As with the previous edition, this edition is not a new features guide or an upgrade book. However, I felt it necessary
to compare areas of behavior where drastic changes occurred, in order to minimize confusion for people who have
used the product for years. New users can simply ignore any references to what SQL Server used to do and
appreciate the product for what it is today.

To help you appreciate SQL Server 2000 even more, the companion CD includes a complete reference to the
Transact-SQL language, including all the system stored procedures. Although this reference is a Microsoft Word
document, it is actually intended for use on line; many of the elements of a syntactic construct contain links to the
further description of that element. When using the document on line, you can simply click on the colored element and
you'll be repositioned at the document with more details about that element. I am grateful to Frank Pellow for sharing
this document. He has included his e-mail address in the document and invites comments and suggestions from
readers.

Now comes the part I've looked forward to the most—thanking all the wonderful people whose enthusiasm and
generosity have made writing this book not only possible but even a pleasure.
This document is created with the unregistered version of CHM2PDF Pilot

First on my list is you, the readers. Thank you, thank you, thank you for reading what I have written. Thank you to
those who have taken the time to write to me about what you thought of the book and what else you want to learn
about. I wish I could answer every question in detail. I appreciate all your input, even when I'm unable to send you a
complete reply.

Thanks to Ron Soukup for writing the first edition and for giving me his "seal of approval" for taking on the
subsequent editions. Thanks to my former editor at SQL Server Professional Journal, Karen Watterson, for
opening so many doors for me, including passing on my name to the wonderful people at Microsoft Press.

The SQL Server development team at Microsoft has been awesome. David Campbell's assistance was invaluable,
once again; in fact, I wouldn't sign the contract to write this new edition until he promised me that he'd still be available
to answer my questions. In this edition, I went into more detail about the query processor and optimizer, and I am
indebted to Lubor Kollar for his enthusiastic support. He actually seemed glad to see me when I stopped by his
office, and he made me feel as if he valued my questions. Thank you, Lubor. Hal Berenson made valuable
contributions to the sections on query processing. Most of the flowcharts in Chapter 15 are from his TechEd
presentation on query processor internals. As with the previous edition, Hal was also my primary resource for the
update of the history section in Chapter 1. Thank you, Hal. Greg Smith, Gert Drapers, Euan Garden, Sameet
Agarwal, Paul Randal, Eric Christensen, Cesar Galindo-Legaria, Wei Xiao, Michael Kung, and Giorgia Paolini
opened their doors to me and responded to my (sometimes seemingly endless) e-mails. Goetz Graefe, Alan Brewer,
Michael Zwilling, Darren Gehring, and Dan Meacham also offered valuable technical insights and information. I owe
very special thanks to Fadi Fakhouri for the time he spent helping me understand the intricacies of SQL Server's
full-text processing capabilities. Other members of the development team, including Damien Lindauer, Ganapathy
Krishnamoorthy, Siva Raghupathy, Srikumar Rangarajan, Balaji Rathakrishnan, Sam Smith, Peter Carlin, Beysim
Sezgin, Steve Lindell, Alazel Acheson, Susan Su, and Dana Burnell provided answers to my e-mails; I hope they all
know how much I appreciated every piece of information I received.

I am also indebted to Steve Dybing and Bob Ward, who aren't really on the development team per se but whose
patience and quick responses I appreciated more than I can say. And many thanks to Arvind Krishnan, who stepped
to the plate in the bottom of the ninth and knocked the ball out of the park with all his help on the last chapter. If not
for him, we might still be in extra innings. (I apologize to international readers and to the translators for the very
American metaphor here.)

I owe more than just simple thanks to Richard Waymire for being a friend as well as a technical resource on the
development team. Even though I don't cover most of his special areas in the book, he was unfailingly generous with
his time and answers and quicker than anybody else in his response to my e-mailed questions. While writing this
edition, I almost felt like a part of the development team myself. This was due not only to the generosity of the
developers but also, in no small part, to Leona Lowry for finding me office space in the same building as most of the
team and allowing interoffice mail to be sent to me through her. Thank you, Leona. The welcome you gave me was
much appreciated.

I received tremendous support from Ryan Trout and Jeff Ressler of the SQL Server marketing team, who provided
advance copies of whitepapers and sample exercises for exploring new features, as well as T-shirts and toys bearing
the SQL Server logo. Ryan also arranged for the loan of a Toshiba laptop, loaded with Windows 2000 Server, so I
could run my tests on a state-of-the-art machine. I am very grateful to Toshiba for their generous assistance. Thanks,
Ryan, and thank you, Toshiba.

I found that other SQL Server trainers were the best help in digging into certain new behaviors and trying to figure
out what was really happening. Thanks to Glenn Christiansen for posing so many puzzling questions and volunteering
lots of technical insights. Writing this book wouldn't have been nearly so much fun without his help. Thanks also to
Kimberly Tripp-Simonnet, Don Vilen, and Arthur Siu for many valuable suggestions on how to explain new
behaviors. My editors at Microsoft Press deserve thanks also. David Clark, my acquisitions editor, got the ball rolling.
Rebecca McKay, the project editor, made sure the chapters kept coming, and my terrific technical editors—Marc
Young and Jean Ross—made sure those chapters turned out well. But, of course, Marc and Jean couldn't do it alone,
so I'd like to thank the rest of the editorial team, including manuscript editor Ina Chang. I know you worked endless
This document is created with the unregistered version of CHM2PDF Pilot

hours to help make this book a reality.

Some people might think that working as an independent trainer and writer can get lonely because I don't have
coworkers to chat with every day in the office. But I've actually got something even better. As a SQL Server MVP, I
work with over a dozen other SQL Server professionals to provide online support on the public newsgroups in the
SQL Server forums. We share a private newsgroup with several Microsoft Support Engineers who serve as our
interface with Microsoft, and through this newsgroup as well as various conferences, I have gotten to know this group
personally. Boundless gratitude is due Itzik Ben-Gan for his amazing generosity, both professional and personal. In
fact, his detailed scrutiny of the previous edition of the book earned him his role as a technical advisor of this edition.
Thank you from the bottom of my heart, Itzik. I really couldn't have done this without you. Roy Harvey, Brian Moran,
and Wayne Snyder offered much needed moral support whenever I asked for it, and more importantly, when I didn't.
B. P. Margolin was extremely generous with his comments on the previous edition and also provided the stored
procedures for estimating table and index sizes, which are on the companion CD. To you guys and to the rest of the
MVP crew—Neil Pike, Tibor Karaszi, Ron Talmadge, Tony Rogerson, Bob Pfeiff, Fernando Guerrero, Umachandar
Jayachandran, Gianluca Hotz, Steve Robertson, Trevor Dwyer, Darren Green, Russell Fields, and our Microsoft
support buddies Shawn Aebi, Steve Dybing, and Paul Wehland—I'd like to say that being a part of your team has
been the greatest honor and privilege of my professional life. I could not imagine working with a more dedicated,
generous, and talented group of people.

Most important of all, my family continues to provide the rock-solid foundation I need to do the work that I do. My
husband Dan continues to be the guiding light of life and my inspiration for doing a job well. My daughter Melissa,
who has just returned from a year of world travel as a Watson Foundation post-graduate fellowship recipient,
continues to be a role model for me as she expands her horizons and decides what she wants to learn about next. My
three sons Brendan, Rickey, and Connor, who are growing up to be generous, loving, and compassionate young men,
have also recently decided to follow in my footsteps. Brendan is becoming a writer, although unlike me he is writing
fiction. I eagerly await each new chapter of the novel he has in progress. Rickey learned to program in half a day this
past summer, and after less than a month is writing more complex games than I ever wrote. And Connor is writing
stories and acting in plays, which is not too different from what I do when teaching classes. My boys have been a
source of countless hugs and kisses and I feel truly blessed to have them in my life.

Kalen Delaney, 2000

This document is created with the unregistered version of CHM2PDF Pilot

34

Preface to the Second Edition

I'll confess right here. I'm a SQL Server junkie. I want to know everything I can about how SQL Server works,
exactly what the internal data structures look like, and what every bit in every status field means in the system tables.
A full list of all the undocumented trace flags and DBCC commands would be my idea of the perfect birthday present.

I also feel like I was born to be a teacher. No matter how late I've been working or how hard it is to get myself out
of bed in the morning, when I stand in front of a classroom, I'm in my element. I love sharing what I know about SQL
Server, and I get even more excited when I have students who want to know about things that I've never even thought
about.

When I was asked to write the update for Ron Soukup's masterpiece, my first thought was that no one besides Ron
could do the topic justice. But then I decided to look at the challenge of the book as the ultimate training assignment,
with thousands of students in the classroom. If I was going to write any book on SQL Server 7, this would be the
one. The topics it covers are all the ones that I love learning about and sharing. The topics it doesn't talk about are the
ones that I haven't spent too much time with and don't really consider myself an expert on.

In Ron's preface to the first edition, he likens his love for SQL Server to his love for a child. I'll go one step further.
Writing this book was like carrying and delivering a child. Not too stressful at first (I never did have much morning
sickness) and lots of eager anticipation. But during the last few months, I felt like it would go on forever. It would
never end! But now it has, and you have this book in your hands and are reading it.

Ron's preface mentions that this book is not an introductory treatise on SQL Server, and I want to emphasize the
same point. This book delves deep into the internal workings of SQL Server and expects that you, the reader, have
some experience with the SQL language and SQL Server already, as well as a fair amount of general technical
sophistication.

This book, as the name implies, deals with the way SQL Server works on the inside. SQL Server 7 includes a
wealth of new peripheral tools, but you'll have to get the details about them in other books. This book doesn't talk
about client programming interfaces, heterogeneous queries, or replication. There are also some internal operations
that aren't discussed, such as security and database backup and restore. I had to draw the line somewhere, or the
book would have been twice as long and I would still be writing it and avoiding calls from my editors asking when the
next chapter would be arriving. I hope you find value here, even if I haven't covered every single topic that you would
like to know about. You can let me know what you'd like to learn more about, and maybe there'll be a second
volume to this book one day. Or maybe I can refer you to other books or whitepapers. You can contact me through
my Web site, at http://www.InsideSQLServer.com.

My editors insisted that this second edition wasn't supposed to be a new features guide or an upgrade book.
However, I knew that lots of people who have used SQL Server for years would be reading this, and I felt it
necessary to compare areas of behavior where drastic changes occurred. New users can just ignore any references to
what SQL Server used to do and appreciate all the wonderful things it does now.

To help you appreciate some of the wonderful features even more, the companion CD includes a whitepaper section,
which serves as a complete reference to the Transact-SQL language, including all the system stored procedures.
Although this is a Microsoft Word document, it is actually intended for use on line, as many of the elements of a
syntactic construct contain links to the further description of that element. When using the document on line, you can
simply click on the colored element and be repositioned at the document with more details describing that element. I
am grateful to Frank Pellow for sharing this document. He has included his e-mail address within the document and
invites comments and suggestions from readers.
This document is created with the unregistered version of CHM2PDF Pilot

Now comes the part I have looked forward to the most (besides holding a published book in my hands). I get to
thank all the wonderful people whose enthusiasm and generosity have made writing this book not only possible but
almost a pleasure.

Thanks to Ron Soukup, for writing the first edition, and giving me his "seal of approval" for undertaking the second
edition. Thanks to my editor at SQL Server Professional Journal, Karen Watterson, for opening so many doors for
me, including passing my name on to the wonderful people at Microsoft Press.

The SQL Server development team at Microsoft has been awesome. David Campbell can never, ever be thanked
enough, as far as I'm concerned. His enthusiasm for the book seemed to exceed my own at times, and his technical
knowledge and clear explanations were priceless, as well as seemingly limitless. Goetz Graefe, Sameet Agarwal,
Mike Habben, Wei Xiao, Nigel Ellis, Eric Christensen, Ross Bunker, Alazel Acheson, Lubor Kollar, and Shaun
Cooper opened their doors to me and responded to (sometimes seemingly endless) e-mails. Cesar Galindo-Legaria,
Peter Byrne, Greg Smith, Michael Zwilling, Jeff Alger, and Alan Brewer also offered valuable technical insights and
information. Other members of the development team provided answers to e-mails, and I hope they all know how
much I appreciated every piece of information I received. I am also indebted to Joe Marler and Bob Ward, who
aren't really on the development team per se, but the technical information they were willing to share was just as
valuable and their pride in SQL Server and dedication to the user community was evident in every e-mail I received
from them.

A special thanks to several Microsoft program managers: Richard Waymire, for being a friend as well as a technical
resource; David Marshall, for helping pull so many pieces together for the general architecture section in Chapter 3;
Bob Vogt, who never seemed to tire of sharing all he could about his special projects, the Index Tuning Wizard, and
SQL Server Profiler; and Hal Berenson, who was a tremendous help on the update of the history section in Chapter 1
.

I received tremendous support from Ryan Trout and Tom Kreyche of the SQL Server marketing team, who
provided advance copies of whitepapers, sample exercises for exploring new features, as well as T-shirts and toys
bearing the SQL Server logo. Thank you so much for everything, especially your big smiles every time I stopped by.

Ryan Labrie made it possible for me to start down the long road of learning about SQL Server 7 shortly after the
Beta 1 release by inviting me to help develop training materials for SQL Server 7. Rick Byham traveled down that
long road with me as we worked together to produce the course, and we were able to spend every day for months
doing nothing but learning about and playing with this powerful new version. I am grateful for that opportunity.

I found that other SQL Server trainers were the best help in digging into certain new behaviors and trying to figure
out experientially what was really happening. Thanks to Glenn Christiansen for posing so many puzzling questions and
Kimberly Tripp-Simonnet for wonderful ideas about how to explain things. Don Vilen deserves special thanks for
trying to talk me out of writing this book; he said it would be much too much work. You were right, Don, but I did it
anyway.

My editors at Microsoft Press deserve thanks also. Ben Ryan, my acquisitions editor, got the ball rolling. Michelle
Goodman, the project editor, made sure the chapters kept coming, and my terrific technical editor, Dail Magee Jr.,
made sure they turned out well. But of course, Dail couldn't do it alone so I'd like to thank the rest of the editorial
team, including many of you whom I never met. I know you worked endless hours to help make this book a reality.

Some people might think that working as an independent consultant and trainer would get lonely, as I don't have
coworkers to chat with every day in the office. But I've got something even better. As a SQL Server MVP, I work
with a dozen other SQL Server professionals to provide online support on the public newsgroups in the SQL Server
forums. We share a private newsgroup with two Microsoft Support Engineers who serve as our interface with
Microsoft, and through this newsgroup and yearly conferences and summits, I have gotten to know this group
personally. Endless gratitude is due Neil Pike for his amazing generosity, both professional and personal. Neil, you are
truly an inspiration. Roy Harvey and Brian Moran offered much needed moral support whenever I asked for it, and
more importantly, when I didn't. To you three, and to the rest of the MVP crew: Tibor Karaszi, Ron Talmadge,
Gianluca Hotz, Tony Rogerson, Bob Pfeiff, Steve Robertson, Trevor Dwyer, Russell Fields, and our Microsoft
This document is created with the unregistered version of CHM2PDF Pilot

support buddies, Shawn Aebi and Paul Wehland, I'd like to let you know that being a part of your team has been the
greatest honor and privilege of my professional life. I could not imagine working with a more dedicated, generous, and
talented group of people.

There's another group of people without whose support and encouragement this book would have never seen the
light of day. My husband Dan never faltered once in his absolute confidence in me and his endless encouragement.
My multitalented daughter Melissa, who will be graduating from Tulane University this spring, has always been an
awesome role model for me. She's someone who can make up her mind to do something or learn something and then
just do it. My three sons Brendan, Rickey, and Connor, who are growing up to be generous, loving, and
compassionate young men, have been the source of countless hugs and kisses and undreamed of patience and
acceptance. I am truly blessed. It's done, guys, and now we're off to DisneyWorld!

Kalen Delaney, 1999

This document is created with the unregistered version of CHM2PDF Pilot

34

Preface to the First Edition

For me, Microsoft SQL Server has been a labor of love. I lived and breathed this product for years, and I look at it
not unlike how I look at my children—with immense love and pride. I've helped nurture the product from its infancy,
through some very tough years, to its current success. I've lost many nights' sleep, upset that some customer was
disappointed with it. Fortunately, I've also had many more thrills and moments of rejoicing—like when a customer
speaks glowingly of SQL Server or when we win another award. Yes, the analogy to a child is the closest thing I can
think of to describe my feelings about SQL Server. Like people who have always thought that one day they'd write
the Great American Novel, I felt a need to finally put down in words some hopefully unique knowledge and opinions
I have regarding SQL Server.

This book is not an introductory treatise on SQL Server, although it does include one introductory chapter (Chapter
2). In Chapter 1, I discuss the history of the product (which I lived), from SQL Server's inception and partnership
with Sybase to its current success. But beyond these early chapters, the book is very detailed and written for those
who want to dig deeply into SQL Server. This book is a suitable read for those who have worked with SQL Server
for years already. It will also be of interest to experienced database professionals new to SQL Server who are
developing or considering new development projects.

In this book, I focus on the "back-end" capabilities of the database engine. Topics include choosing appropriate
hardware for SQL Server, effective use of the SQL language, writing correct queries, common mistakes, consistency
and concurrency tradeoffs, data storage and structures, locking, and scrollable cursors. Performance considerations
are covered throughout the book. The final chapters (Chapters 14 and 15) discuss specific performance issues, but
because they assume that you've gained much knowledge from reading earlier chapters in the book, I strongly
discourage you from jumping directly to those chapters.

When I planned the book, I felt it was necessary to describe how SQL Server works before going into the product's
functional capabilities. But that presents a "chicken-or-the-egg" problem: what should I introduce first? I decided to
include architectural information near the beginning of the book—Chapter 3 provides an in-depth discussion of how
SQL Server works. If you are already familiar with the functional aspects of SQL Server, I think you'll benefit most
from this chapter. If you are new to SQL Server, I encourage you to go back and read Chapter 3 again in a couple
months after you've gained more familiarity with the product. I think you'll get a lot more out of it. Initially, simply try
to absorb just the basics.

I've included little discussion of client programming issues, such as which development tools to use or details about
ODBC programming. If you need information about these topics, see the Suggested Reading section near the end of
the book. And I'm sorry to say that there is little discussion of the replication capabilities of SQL Server. (This is also
a little embarrassing because I recently took charge of managing the Replication Unit within the SQL Server
development team.) But, like software, this thing had to eventually ship, and replication ended up on the "cut" list. I
promise that we're doing so many exciting new things in replication for the next version that I knew I would have to
almost completely rewrite any replication section before long anyway. (And a very good whitepaper about SQL
Server replication capabilities is included on the companion CD.)

This book has a healthy smattering of my personal opinions about many issues in software and database
development. Software development is still at least as much an art as a science, and there are different schools of
thought and different approaches. Healthy, even if heated, debate is one of the things that make software development
so much fun. I elected to be pretty candid and direct with my opinions in many places, rather than keep the book
bland and neutral. Some people will no doubt disagree strongly with some of my opinions. That's fine. I respect your
right to disagree. But I do hope you can read and accept the book for its overall merits, even if you disagree with a
few topics. And, of course, the opinions I present here are mine alone and not necessarily those of Microsoft.
This document is created with the unregistered version of CHM2PDF Pilot

Now I have many people to thank. Without the product, there would be no book. So my thanks must first be to my
fellow SQL Server "old-timers," who bootstrapped the product. Working with you was an amazing, unforgettable
experience. We worked 80 hours a week far more often than not, but we loved it. No one deserves more credit for
SQL Server's success than Rick Vicik. I'm sure Rick is a genius, but he has the rare trait among those with genius IQs
of being perfectly pragmatic. He's a doer, and no one works harder or more passionately. He led the development
team by sheer example. And what a team. Without the other old-timers like Mike Habben, Lale Divringi, Peter
Hussey, Chris Cassella, Chris Moffatt, Kaushik Chodhury, Dave Fyffe, Don Reichart, Craig Henry, Jack Love, Karl
Johnson, Ted Hart, Gavin Jancke, Dan Tyack, Trish Millines, Mike Ervin, Mike Byther, Glen Barker, Bryan Minugh,
and Heidy Krauer, there simply would be no Microsoft SQL Server today. And the early documentation team of
Lesley Link and Helen Meyers, and the original marketing team of Dwayne Walker, Gary Voth, and Dan Basica,
made SQL Server a product, not just software. It took this unique and special blend of people to make this product
and to make it viable. There are, of course, many others of great talent who have joined the SQL Server team during
the last few years. But if not for the extraordinary efforts of the original, very small team, there would be no SQL
Server now. Be proud and always remember what we did together.

We needed and were backed up by a great customer support team, anchored by Andrea Stoppani, Gary Schroeder,
Rande Blackman, Joe Marler, Vaqar Pirzada, Damien Lindauer, and James McDaniel (several of whom are now on
the SQL Server development team). You often went above and beyond the call of duty. Thank you.

Turning toward the book, I want to extend special thanks to Lesley Link. Lesley did the huge job of editing this book
entirely in her "spare time" (which meant many late nights and early Sunday mornings). She really committed to editing
this book because of the pride and ownership she, too, feels in the product. But while I wrote the bulk of the book
more or less on a sabbatical leave, Lesley edited while she continued in her more than full-time job as the manager of
the SQL Server documentation team. There is no better editor in the business. And thanks to her production team,
especially Steven Fulgham, Christine Woodward, Pat Hunter, and Rasa Raisys, for helping prepare the book.

Thank you to those who provided invaluable technical reviews of various sections, especially to Jim Gray, Rick
Vicik, Mike Habben, Peter Hussey, Don Vilen, Dave Campbell, and Lale Divringi. The mistakes that no doubt
remain are mine alone.

Special thanks go to Gary Schroeder, who authored the Microsoft internal documentation for the on-disk data
structures that I made use of when describing disk and table formats. And thanks to Betty O'Neil, who revamped and
reorganized much of the internal design documentation that I used as a resource when writing.

Finally, thanks to the staff at Microsoft Press: David, Lisa, John, and those of you I never met but knew were there,
who guided me through the publication process.

This is the first book I've written. I've shipped a lot more software than books. I've learned there are a lot of
similarities. Like the last 10 percent takes 90 percent of the time. And that when it's done, you always have this
nagging feeling that you'd like to do some part over and could do it much better the second time around. After all the
demands and grief I gave them over the years, I know that the documentation team that formerly reported to me has
taken great delight in watching me write and seeing me struggle to complete this work. And just as software is never
perfect, neither is this book. I've always found that fact painful in shipping software—I always want it to be perfect.
And I find it painful now. I wish this book were perfect, and I know it's not.

But, as I always remind my development team, "Shipping is a feature." So, after a long labor, I have shipped.

Ron Soukup, 1997

This document is created with the unregistered version of CHM2PDF Pilot

34

System Requirements for the

Companion CD-ROMs
System
 PC with an Intel or compatible Pentium 166-MHz or higher processor
 Microsoft Windows NT Server 4.0 with Service Pack 5 or later, Windows NT Server 4.0 Enterprise Edition
with Service Pack 5 or later, Windows 2000 Server, Windows 2000 Advanced Server, or Windows 2000
DataCenter Server operating system
 Minimum of 64 MB of RAM (128 MB or more recommended)

Hard drive space

 95–270 MB for database server; approximately 250 MB for typical installation
 50 MB minimum for Analysis Services; 130 MB for typical installation
 80 MB for Microsoft English Query (supported on Windows 2000 operat-ing system but not logo certified)

Other
 Microsoft Internet Explorer 5.0 or later
 Microsoft Word 97 or later
 Microsoft Excel 97 or later
 CD-ROM drive
 VGA or higher resolution monitor
 Microsoft Mouse or compatible pointing device
This document is created with the unregistered version of CHM2PDF Pilot

34

Part I

Overview
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 1

The Evolution of Microsoft SQL

Server: 1989 to 2000
In 1985, Microsoft and IBM announced "a long-term joint development agreement for development of operating
systems and other systems software products." This announcement was the beginning of OS/2, a successor to the
Microsoft MS-DOS operating system. OS/2 would be a more complete and robust operating system. It would
exploit the powerful new personal computers based on the Intel 80286 processor. And it would allow multitasking
applications, each with its own address space and each running in the safe ring 3 of the Intel four-ring protection
scheme of the 80286. Machines sold in 1986 would be vastly more powerful than the original IBM PC (an Intel
8088-based machine) of just a couple years earlier, and OS/2 would signal a new age in harnessing this power. That
was the plan.

OS/2 was formally announced in April 1987, with shipment promised by the end of the year. (OS/2 version 1.0 was
released to manufacturing on December 16, 1987.) But shortly after the joint declaration, IBM announced a special
higher-end version of OS/2 called OS/2 Extended Edition. This more powerful version would include the base OS/2
operating system plus an SQL RDBMS called OS/2 Database Manager. OS/2 Database Manager would be useful
for small applications and would be partially compatible (although not initially interoperable) with DB/2, IBM's
flagship MVS mainframe database, and with the lesser-used SQL/DS, which ran on slightly smaller mainframe
computers using the VM or VSE operating systems. OS/2 Database Manager would also include Systems Network
Architecture (SNA) communications services, called OS/2 Communications Manager. As part of its sweeping
Systems Application Architecture (SAA), IBM promised to make the products work well together in the future.
(OS/2 Database Manager later evolved into today's DB2/2.)

But if IBM could offer a more complete OS/2 solution, who would buy Microsoft OS/2? Clearly, Microsoft needed
to come up with an answer to this question.
This document is created with the unregistered version of CHM2PDF Pilot

34

SQL Server: The Early Years

In 1986, Microsoft was a mere $197-million-per-year business with 1153 employees. (Ten years later, Microsoft
had revenues of nearly $6 billion and almost 18,000 employees.) Microsoft's products were entirely desktop focused,
and the main bread-and-butter product was MS-DOS. Client/server computing was not yet in the vernacular of
Microsoft or the computer industry. Data management on PCs was in its infancy. Most people who kept data on their
PCs used the wildly popular Lotus 1-2-3 spreadsheet application to keep lists (although many were discovering the
limitations of doing so). Ashton-Tate's dBASE products (dBASE II and the recently released dBASE III) had also
become popular. Although a few other products existed, such as MicroRim's Rbase and a relatively new product
from Ansa Software called Paradox, Ashton-Tate was clearly king of the PC data products. In 1986, Microsoft had
no database management products. (Beginning in 1992, Microsoft would go on to achieve tremendous success in the
desktop database market with Microsoft Access and Microsoft FoxPro.)

IBM's Database Manager wasn't in the same category as products such as dBASE, Paradox, and Rbase. Database
Manager was built to be a full-featured database (with atomic transactions and a full SQL query processor), more
similar to traditional minicomputer-oriented or mainframe-oriented systems such as IBM's DB/2, or Oracle, or
Informix. Microsoft needed a database management system (DBMS) product of the same caliber, and it needed one
soon.

Microsoft turned to Sybase, Inc., an upstart in the DBMS market. Sybase hadn't yet shipped the first commercial
version of its DataServer product (which it would do in May 1987 for Sun workstations running UNIX). Although
certainly not a mainstream product, the prerelease version of DataServer had earned a good reputation for delivering
innovative new capabilities, such as stored procedures and triggers, and because it had been designed for a new
paradigm in computing: client/server.

As is true in all good business exchanges, the deal between the two companies was a win-win situation. Microsoft
would get exclusive rights to the DataServer product for OS/2 and all other Microsoft-developed operating systems.
Besides getting royalties from Microsoft, Sybase would get credibility from Microsoft's endorsement of its technology.
Even more important, Sybase would gain a beachhead among the anticipated huge number of personal computers that
would be running the new OS/2 operating system.

Because the transaction-processing throughput of these OS/2 systems wasn't expected to be high, Sybase could use
the systems to seed the market for future sales on the more powerful UNIX system. Microsoft would market the
product in higher volumes than Sybase could; it simply wasn't economically feasible for Sybase's direct sales force to
deliver what would essentially be the first shrink-wrapped release of a full-fledged, blood-and-guts database to PC
customers. Higher volumes would help Sybase win more business on UNIX and VMS platforms. On March 27,
1987, Microsoft president Jon Shirley and Sybase cofounder and president Mark Hoffman signed the deal.

In the PC database world, Ashton-Tate's dBASE still had the reputation and the lion's share of the market, even if
dBASE and Sybase DataServer offered extremely different capabilities. To gain acceptance, this new,
higher-capability DBMS from Microsoft (licensed from Sybase) would need to appeal to the large dBASE
community. The most direct way to do that, of course, was to get Ashton-Tate to endorse the product—so Microsoft
worked out a deal with Ashton-Tate to do just that.

In 1988, a new product was announced with the somewhat clumsy name Ashton-Tate/Microsoft SQL Server.
Although not part of the product's title, Sybase was prominent in the product's accompanying information. This new
product would be a port of Sybase DataServer to OS/2, marketed by both Ashton-Tate and Microsoft. Ashton-Tate
had pledged that its much-anticipated dBASE IV would also be available in a server edition that would use the
dBASE IV development tools and language as a client to develop applications (for example, order-entry forms) that
would store the data in the new SQL Server product. This new client/server capability promised to give dBASE new
This document is created with the unregistered version of CHM2PDF Pilot

levels of power to support more than the handful of concurrent users that could be supported by its existing
file-sharing architecture.

Ashton-Tate, Microsoft, and Sybase worked together to debut SQL Server on OS/2. (This was the first use of the
name SQL Server. Sybase later renamed its DataServer product for UNIX and VMS Sybase SQL Server. Today,
Sybase's database server is known as Sybase Adaptive Server.)

The first beta version of Ashton-Tate/Microsoft SQL Server shipped in the fall of 1988. Microsoft made this
prerelease version available at nominal cost to developers who wanted to get a head start on learning, developing for,
or evaluating this new product. It shipped in a bundle known as the NDK (network development kit) that included all
the necessary software components (provided that you were developing in C) for getting a head start on building
networked client/server applications. It included prerelease versions of SQL Server, Microsoft LAN Manager, and
OS/2 1.0.
This document is created with the unregistered version of CHM2PDF Pilot

34

Ron's Story
In 1988, Ron Soukup was working for Covia, the United Airlines subsidiary that provided the Apollo Reservation
System and related systems for airport and travel agency use. He had spent the previous five years working with the
new breed of relational database products that had appeared on the minicomputer and mainframe computer scene.
He had worked with IBM's DB/2 running on MVS; IBM's SQL/DS running on VM; and Oracle, Informix, and Unify
running on UNIX. Ron viewed PC databases of the day essentially as toys that were good for storing recipes and
addresses but not much else. He used a PC for word processing, but that was about it. At work, they were beginning
to use more and more LAN-based and PC-based systems, and Ron had begun doing some OS/2 programming.
When he heard of the NDK, with this new SQL Server product that had been mentioned in the trade press, he
ordered it immediately.

The NDK was very much a beta-level product. It didn't have a lot of fit and finish, and it crashed a couple of times a
day. But from practically the first day, Ron knew that this product was something special. He was amazed that he was
using a true DBMS—on a PC!—with such advanced features as transaction logging and automatic recovery. Even the
performance seemed remarkably good. Having used mostly minicomputer and mainframe computer systems, he was
most struck by the difference in PC response time. With the bigger systems, even a simple command resulted in an
inevitable delay of at least a couple of seconds between pressing the Enter key and receiving a reply. PCs seemed
almost instantaneous. Ron knew PCs were fast for local tasks such as word processing, but this was different. In this
case, at one PC he entered an SQL command that was sent over the network to a server machine running this new
SQL Server product. The response time was a subsecond. He had never seen such responsiveness.

Ron's initial kick-the-tires trial was encouraging, and he received approval to test the product more thoroughly. He
wanted to get a feel for the types of applications and workloads for which this interesting new product might be used.
For this, Ron wanted more substantial hardware than the desktop machine he originally tested on (a 10MHz 286
computer with 6 MB of memory and a 50-MB hard drive). Although SQL Server ran reasonably well on his desktop
machine, he wanted to try it on one of the powerful new machines that used the Intel 80386 processor. Ron was able
to procure a monster machine for the day—a 20-MHz 386 system with 10 MB of memory and two 100-MB disk
drives—and was the envy of his division!

In 1987, Ron and a colleague at Covia had developed some multiuser database benchmark tests in C to help them
choose a UNIX minicomputer system for a new application. So Ron dusted off these tests and converted the
embedded C to the call-level interface provided in SQL Server (DB-Library) and ported these benchmarks to the
PC. Ron hoped that SQL Server would handle several simultaneous users, although he didn't even consider that it
could come close to handling the 15 to 20 simulated users they had tried in the earlier minicomputer tests. After many
false starts and the typical problems that occur while running an early beta of a version 1.0 product, he persevered
and got the test suite—2000 lines of custom C code—running on a PC against the beta version of SQL Server.

The results were amazing. This beta version of SQL Server, running on a PC that cost less than $10,000, performed
as well and in many cases better than the minicomputer systems that they had tested a few months earlier. Those
systems cost probably 10 times as much as the PC that Ron was using, and they needed to be managed by a
professional UNIX system administrator. Ron knew the industry was in for a big change.

In May 1989, Ashton-Tate/Microsoft SQL Server version 1.0 shipped. Press reviews were good, but sales lagged.
OS/2 sales were far below what had been expected. Most users hadn't moved from MS-DOS to OS/2, as had been
anticipated. And about the only tool available to create SQL Server applications was C. The promised dBASE IV
Server Edition from Ashton-Tate was delayed, and although several independent software vendors (ISVs) had
promised front-end development tools for SQL Server, these hadn't materialized yet.

During the preceding six months, Ron had come to really know, respect, and admire SQL Server, and he felt the
This document is created with the unregistered version of CHM2PDF Pilot

same about the people at Microsoft with whom he had worked during this period. So in late 1989, Ron accepted a
position at Microsoft in the SQL Server group in Redmond, Washington. A few months later, he was running the
small but talented and dedicated SQL Server development team.
This document is created with the unregistered version of CHM2PDF Pilot

34

Kalen's Story
Kalen Delaney saw the first version of Microsoft SQL Server at about the same time that Ron did. She started
working for Sybase Corporation's technical support team in 1987. And in 1988, she started working with a team that
was testing the new PC-based version of Sybase SQL Server, which a company in the Seattle area was developing
for OS/2. Sybase would be providing phone support for customers purchasing this version of the product, and Kalen,
along with the rest of the technical support team, wanted to be ready.

Up to that point, she had never actually worked with PC-based systems and had pegged them as being mere toys.
Sybase SQL Server was an extremely powerful product, capable of handling hundreds of simultaneous users and
hundreds of megabytes of data. Kalen supported customers using Sybase SQL Server databases to manage
worldwide financial, inventory, and human resource systems. A serious product was required to handle these kinds of
applications, and Kalen thought PCs just didn't fit the bill. However, when she saw a PC sitting on a coworker's desk
actually running SQL Server, Kalen's attitude changed in a hurry. Over the next few years, she worked with hundreds
of customers in her product support position and later in her position as Senior Instructor, using Microsoft SQL
Server to manage real businesses with more than toy amounts of data, from a PC.

In 1991, Kalen and her family left the San Francisco Bay Area to live in the Pacific Northwest, but she continued to
work for Sybase as a trainer and courseware developer. A year later, Sybase dissolved her remote position, and she
decided to start her own business. A big decision loomed: should she focus primarily on the Sybase version or the
Microsoft version of SQL Server? Because Microsoft SQL Server would run on a PC that she could purchase at her
local Sears store with low monthly payments, and because she could buy SQL Server software and the required
operating system in one package, the decision was simple. Microsoft had made it quite easy for the sole proprietor to
acquire and develop software and hone his or her skills on a powerful, real-world, relational database system.
This document is created with the unregistered version of CHM2PDF Pilot

34

Microsoft SQL Server Ships

By 1990, the co-marketing and distribution arrangement with Ashton-Tate, which was intended to tie SQL Server to
the large dBASE community, simply wasn't working. Even the desktop version of dBASE IV was quite late, and it
had a reputation for being buggy when it shipped in 1989. The Server Edition, which would ostensibly make it simple
to develop higher-performance SQL Server applications using dBASE, was nowhere to be seen.

As many others have painfully realized, developing a single-user, record-oriented application is quite different from
developing applications for multiple users for which issues of concurrency, consistency, and network latency must be
considered. Initial attempts at marrying the dBASE tools with SQL Server had dBASE treating SQL Server as
though it were an indexed sequential access method (ISAM). A command to request a specific row was issued for
each row needed. Although this procedural model was what dBASE users were accustomed to, it wasn't an efficient
way to use SQL Server, with which users could gain more power with less overhead by issuing SQL statements to
work with sets of information. But at the time, SQL Server lacked the capabilities to make it easy to develop
applications that would work in ways dBASE users were accustomed to (such as browsing through data forward and
backward, jumping from record to record, and updating records at any time). Scrollable cursors didn't exist yet.
NOTE
The effort to get dBASE IV Server Edition working well provided many ideas for how scrollable cursors in a
networked client/server environment should behave. In many ways, it was the prime motivation for including this
feature in SQL Server version 6.0 in 1995, six years later.

Only two years earlier, Ashton-Tate had been king of the PC database market. Now it was beginning to fight for its
survival and needed to refocus on its core dBASE desktop product. Microsoft would launch OS/2 LAN Manager
under the Microsoft name (as opposed to the initial attempts to create only OEM versions), and it needed SQL
Server to help provide a foundation for the development of client/server tools that would run on Microsoft LAN
Manager and Microsoft OS/2. So Microsoft and Ashton-Tate terminated their co-marketing and distribution
arrangements. The product would be repackaged and reintroduced as Microsoft SQL Server.

Microsoft SQL Server version 1.1 shipped in the summer of 1990 as an upgrade to the Ashton-Tate/Microsoft SQL
Server version 1.0 that had shipped in 1989. For the first time, SQL Server was a Microsoft-supported,
shrink-wrapped product, and it was sold through the newly formed Microsoft Network Specialist channel, whose
main charter was to push Microsoft LAN Manager.
NOTE
When version 1.1 shipped, Microsoft didn't see SQL Server as a lucrative product in its own right. SQL Server
would be one of the reasons to buy LAN Manager—that's all.

SQL Server 1.1 had the same features as version 1.0, although it included many bug fixes—the type of maintenance
that is understandably necessary for a version 1.0 product of this complexity. But SQL Server 1.1 also supported a
significant new client platform, Microsoft Windows 3.0. Windows 3.0 had shipped in May 1990, a watershed event
in the computer industry. SQL Server 1.1 provided an interface that enabled Windows 3.0-based applications to be
efficiently developed for it. This early and full support for Windows 3.0-based applications proved to be vital to the
success of Microsoft SQL Server. The success of the Windows platform would also mean fundamental changes for
Microsoft and SQL Server, although these changes weren't yet clear in the summer of 1990.

With the advent of Windows 3.0 and SQL Server 1.1, many new Windows-based applications showed up and
many were, as promised, beginning to support Microsoft SQL Server. By early 1991, dozens of third-party software
products used SQL Server. SQL Server was one of the few database products that provided a Windows 3.0
dynamic link library (DLL) interface practically as soon as Windows 3.0 shipped, which was now paying dividends in
the marketplace. Quietly but unmistakably, Microsoft SQL Server was the leading database system used by
This document is created with the unregistered version of CHM2PDF Pilot

Windows applications. Overall sales were still modest, but until tools beyond C existed to build solutions, sales
couldn't be expected to be impressive. It was the classic chicken-and-egg situation.
This document is created with the unregistered version of CHM2PDF Pilot

34

Development Roles Evolve

Microsoft's development role for SQL Server 1.0 was quite limited. As a small porting team at Sybase moved its
DataServer engine to OS/2 and moved the DB-Library client interfaces to MS-DOS and OS/2, Microsoft provided
testing and project management. Microsoft also developed some add-on tools to help make SQL Server 1.0 easy to
install and administer.

Although a number of sites were running OS/2 as an application server with SQL Server or as a file server with LAN
Manager, few were using OS/2 for their desktop platforms. Before Windows 3.0, most desktops remained
MS-DOS based, with a well-known limit of 640 KB of addressable real memory. After loading MS-DOS, a
network redirector, network card device drivers, and the DB-Library static link libraries that shipped with SQL
Server version 1.0, developers trying to write a SQL Server application were lucky to get 50 KB for their own use.

For SQL Server 1.1, rather than shipping the DB-Library interface that Sybase had ported from UNIX to
MS-DOS, Microsoft wrote its own, from scratch. Instead of 50 KB, developers could get up to 250 KB to write
their applications. Although small by today's standards, 250 KB was a huge improvement.
NOTE
The same source code used for DB-Library for MS-DOS also produced the Windows and OS/2 DB-Library
interfaces. But the MS-DOS RAM cram is what motivated Microsoft to write a new implementation from scratch.
The widespread adoption of Windows 3.0 quickly eliminated the MS-DOS memory issue—but this issue was a real
problem in 1989 and 1990.

With SQL Server 1.1, Microsoft provided client software and utilities, programming libraries, and administration
tools. But the core SQL Server engine was still produced entirely by Sybase; Microsoft didn't even have access to
the source code. Any requests for changes, even for bug fixes, had to be made to Sybase.

Microsoft was building a solid support team for SQL Server. It hired some talented and dedicated engineers with
database backgrounds. But with no access to source code, the team found it impossible to provide the kind of
mission-critical responsiveness that was necessary for customer support. And, again, fixing bugs was problematic
because Microsoft depended entirely on Sybase, which had become successful in its own right and was grappling
with its explosive growth. Inevitably, some significant differences arose in prioritizing which issues to address,
especially when some issues were specific to the Microsoft-labeled product and not to Sybase's product line. Bugs
that Microsoft deemed of highest priority sometimes languished because Sybase's priorities were understandably
directed elsewhere. The situation was unacceptable.

It was a great day in the SQL Server group at Microsoft when, in early 1991, Microsoft's agreement with Sybase
was amended to give Microsoft read-only access to the source code for the purpose of customer support. Although
Microsoft's SQL Server group still couldn't fix bugs, they at least could read the source code to better understand
what might be happening when something went wrong. And they could also read the code to understand how
something was expected to work. As anyone with software development experience knows, even the best
specification is ambiguous at times. There's simply no substitute for the source code as the definitive explanation for
how something works.

As a small group of developers at Microsoft became adept with the SQL Server source code and internal workings,
Microsoft began to do virtual bug fixes. Although they still weren't permitted to alter the source code, they could
identify line-by-line the specific modules that needed to be changed to fix a bug. Obviously, when they handed the fix
directly to Sybase, high-priority bugs identified by Microsoft were resolved much more quickly.

After a few months of working in this way, the extra step was eliminated. By mid-1991, Microsoft could finally fix
This document is created with the unregistered version of CHM2PDF Pilot

bugs directly. Because Sybase still controlled the baseline for the source code, all fixes were provided to Sybase for
review and inclusion in the code. The developers at Microsoft had to make special efforts to keep the source code
highly secured, and the logistics of keeping the source code in sync with Sybase was sometimes a hassle, but all in all,
this was heaven compared to a year earlier. Microsoft's team of developers was becoming expert in the SQL Server
code, and they could now be much more responsive to their customers and responsible for the quality of the product.
This document is created with the unregistered version of CHM2PDF Pilot

34

OS/2 and Friendly Fire

In 1991, Microsoft released SQL Server 1.11, a maintenance release. SQL Server was slowly but steadily gaining
acceptance and momentum—and a long list of ISV supporters. Client/server computing wasn't widely deployed yet,
but new converts appeared every day. Customer satisfaction and loyalty were high, and press reviews of the product
were all favorable. Sales were generally disappointing, but this was hardly a surprise because OS/2 had continued to
be a major disappointment. Windows 3.0, however, was a runaway hit. Rather than move their desktop platforms
from MS-DOS to OS/2, huge numbers of PC users moved to Windows 3.0 instead. OS/2 hadn't become a
widespread operating system as anticipated, and it was now abundantly clear that it never would be.

SQL Server's Limitations and the Marketplace

Microsoft SQL Server 1.11 clearly had a scalability limit. It was a 16-bit product because OS/2 could provide only
a 16-bit address space for applications, and its throughput was hampered by OS/2's lack of some high-performance
capabilities, such as asynchronous I/O. Even though an astonishing amount of work could be performed successfully
with SQL Server on OS/2, there would come a point at which it would simply run out of gas. No hard limit was
established, but in general, SQL Server for OS/2 was used for workgroups of 50 or fewer users. For larger groups,
customers could buy a version of Sybase SQL Server for higher-performance UNIX-based or VMS-based
systems.

This was an important selling point for both Microsoft and Sybase. Customers considering the Microsoft product
wanted to be sure they wouldn't outgrow it. The large number of ISV tools developed for Microsoft SQL Server
worked largely unchanged with Sybase SQL Server, and applications that outgrew OS/2 could be moved quite easily
to a bigger, more powerful, more expensive UNIX system. This relationship still made sense for both Microsoft and
Sybase.

The need for compatibility and interoperability made it especially important for Microsoft SQL Server to be based on
the version 4.2 source code as soon as possible. Furthermore, a major features version hadn't been released since
version 1.0 in 1989. In the rapidly moving PC marketplace, the product was in danger of becoming stale. Customers
had begun to do serious work with Microsoft SQL Server, and new features were in great demand. Microsoft's
version 4.2 would add a long list of significant new features, including server-to-server stored procedures, UNION,
online tape backup, and greatly improved international support that would make SQL Server more viable outside the
United States.

At the same time, Microsoft was working on a new SQL Server version that would sync up with the newest Sybase
product on UNIX, version 4.2. When Microsoft SQL Server 1.0 shipped, Sybase's product was designated version
3.0. They had added some new features deemed necessary for the PC marketplace, such as text and image
datatypes and browse mode. Sybase subsequently shipped version 4.0 for most platforms and version 4.2 on a more
limited basis.

Meanwhile, in May 1991, Microsoft and IBM announced an end to their joint development of OS/2. Clearly, most
customers were voting with their dollars for Windows, not OS/2. Microsoft decided to concentrate on future versions
of Windows and applications for Windows. The announcement, although not a surprise, rocked the industry
nonetheless. Microsoft was well underway in the development of a new microkernel-based operating system that was
internally code-named NT (for "new technology"). This new system was originally envisioned as a future release of
OS/2 and was sometimes referred to as OS/2 3.0. After the termination of joint OS/2 development, the NT project
was altered to include the Windows user interface and the Win32 application programming interface (API), and it
became known henceforth as Microsoft Windows NT.

The first version of Windows NT wasn't expected for two years. Microsoft SQL Server would eventually be moved
This document is created with the unregistered version of CHM2PDF Pilot

to Windows NT—that was a no-brainer. But in the meantime, Microsoft had to continue developing SQL Server on
OS/2, even though OS/2 was now a competitive product for Microsoft; there was no alternative. For the next couple
of years, the SQL Server group got used to getting hit by friendly fire as Microsoft competed vigorously against
OS/2.
This document is created with the unregistered version of CHM2PDF Pilot

34

SQL Server 4.2

Microsoft had been developing SQL Server 4.2 for the forthcoming OS/2 2.0, the first 32-bit version of OS/2. Since
SQL Server 4.2 was also planned to be 32-bit, porting the product from its UNIX lineage would be easier because
memory segmentation issues wouldn't be a concern. In theory, the 32-bit SQL Server would also perform faster.
Many articles comparing 32-bit and 16-bit performance issues appeared in the press, and everyone assumed that the
32-bit environment would bring awesome performance gains (although few articles explained correctly why this might
or might not be true).

The principal performance gain expected would be due to memory addressing. To address memory in the 16-bit
segmented address space of OS/2 1.x, basically two instructions were required: one to load the correct segment and
one to load the memory address within that segment. With 32-bit addressing, the instruction to load the segment was
unnecessary and memory could be addressed with one instruction, not two. Because addressing memory is so
common, some quick calculations showed that the 32-bit version might yield an overall performance increase of
perhaps 20 percent or more, if all other operations were of equal speed.

The 32-Bit Platform: Bigger Isn't Always Better

Many people mistakenly believed that SQL Server needed to be running on a fully 32-bit platform to address more
than 16 MB of memory. Under OS/2 1.x, an application could access a maximum of 16 MB of real memory; it could
access more than 16 MB of virtual memory, but paging would result. In OS/2 2.0, an application could access more
than 16 MB of real memory and avoid paging. This would allow SQL Server to have a larger cache, and getting data
from memory rather than from disk always results in a huge performance gain. However, to the application, all
memory in OS/2 was virtual memory—in both versions. So even the 16-bit version of SQL Server would be able to
take advantage of the ability of OS/2 2.0 to access larger real-memory spaces. A 32-bit version was unnecessary for
this enhancement.

Unfortunately, the early beta versions of OS/2 2.0 were significantly slower than OS/2 1.x, more than offsetting the
efficiency in addressing memory. So rather than a performance gain, users saw a significant loss of performance in
running Microsoft SQL Server 1.1 and preliminary builds of the 32-bit SQL Server 4.2 on OS/2 2.0.

OS/2 2.0 Release on Hold

Suddenly, the plans for the release of OS/2 2.0 by the end of 1991 were suspect. In fact, it was unclear whether
IBM could deliver version 2.0 at all. At any rate, it appeared doubtful that OS/2 would ship any sooner than the end
of 1992. The decision became clear—Microsoft would move SQL Server back to a 16-bit implementation and target
it for OS/2 1.3.

Reworking back to 16-bit would cost the Microsoft developers about three months, but they had little choice. In the
meantime, another problem appeared. IBM shipped OS/2 1.3, but this version worked only for its brand of PCs.
Theoretically, other computer manufacturers could license OS/2 from Microsoft and ship it as part of an OEM
agreement. However, the demand for OS/2 had become so small that most OEMs didn't ship it; as a result, buying
OS/2 for other PCs was difficult for customers. For the first time, despite the seeming incongruity, Microsoft
produced a shrink-wrapped version of OS/2 version 1.3, code-named Tiger. Tiger shipped in the box with Microsoft
SQL Server and Microsoft LAN Manager, lessening the problem that the product was essentially targeted to a dead
operating system.

Version 4.2 Released

This document is created with the unregistered version of CHM2PDF Pilot

Microsoft SQL Server version 4.2 entered beta testing in the fall of 1991, and in January 1992, Microsoft CEO Bill
Gates (with Sybase's Bob Epstein sharing the stage) formally announced the product at a Microsoft SQL Server
developers' conference in San Francisco. Version 4.2 truly had been a joint development between Microsoft and
Sybase. The database engine was ported from the UNIX version 4.2 source code, with both Microsoft and Sybase
engineers working on the port and fixing bugs. In addition, Microsoft produced the client interface libraries for
MS-DOS, Windows, and OS/2, and for the first time it included a Windows GUI tool to simplify administration.
Source code for the database engine was merged back at Sybase headquarters, with files exchanged via modem and
magnetic tape.

Microsoft SQL Server version 4.2 shipped in March 1992. The reviews were good and customer feedback was
positive. As it turned out, the source code for the database engine in this version was the last code that Microsoft
received from Sybase (except for a few bug fixes exchanged from time to time).

After the product shipped, the Big Question for 1992 was, "When will a 32-bit version of Microsoft SQL Server be
available?" The issue of 32-bitness became an emotional one. Many people who were the most strident in their
demands for it were unclear or confused about what the benefits would be. They assumed that it would automatically
have a smaller footprint, run faster, address more memory, and generally be a much higher-performing platform. But
the internal work with a 32-bit version for OS/2 2.0 had shown that this wouldn't necessarily be the case.
This document is created with the unregistered version of CHM2PDF Pilot

34

SQL Server for Windows NT

Contrary to what many people might assume, Microsoft senior management never pressured the SQL Server
development team to not develop a full 32-bit version for OS/2. One of the joys of working for Microsoft is that
senior management empowers the people in charge of projects to make the decisions, and this decision was left with
Ron Soukup.

In early 1992, however, the development team faced some uncertainty and external pressures. On one hand, the
entire customer base was, by definition, using OS/2. Those customers made it quite clear that they wanted—indeed,
expected—a 32-bit version of SQL Server for OS/2 2.0 as soon as IBM shipped OS/2 2.0, and they intended to
remain on OS/2 in the foreseeable future. But when OS/2 2.0 would be available was unclear. IBM claimed that it
would ship by the fall of 1992. Steve Ballmer, Microsoft senior vice president, made a well-known pledge that he'd
eat a floppy disk if IBM shipped the product in 1992.

The SQL Server team was now under pressure to have a version of SQL Server running on Windows NT as soon
as possible, with prerelease beta versions available when Windows NT was prereleased. The SQL Server team
members knew that Windows NT was the future. It would be the high-end operating system solution from Microsoft,
and from a developer's perspective, Windows NT would offer a number of technical advantages over OS/2, including
asynchronous I/O, symmetric multiprocessing, and portability to reduced instruction set computing (RISC)
architectures. They were champing at the bit to get started.

Although Microsoft had decided in 1991 to fall back to a 16-bit version of SQL Server, Microsoft's developers had
continued to work on the 32-bit version. By March 1992, just after shipping version 4.2, they saw that both the
16-bit and 32-bit versions ran slower on the beta versions of OS/2 2.0 than the 16-bit version ran on Tiger (OS/2
1.3). Perhaps by the time OS/2 2.0 actually shipped, it might run faster. But then again, it might not—the beta updates
of OS/2 2.0 didn't give any indication that it was getting faster. In fact, it seemed to be getting slower and more
unstable.

For a product with the scope of SQL Server, there's no such thing as a small release. There are big releases and
bigger releases. Because resources are finite, the developers knew that working on a product geared toward OS/2
2.0 would slow down the progress of Windows NT development and could push back its release. Adding more
developers wasn't a good solution. (As many in the industry have come to learn, adding more people is often the
cause of, and seldom the solution to, software development problems.) Furthermore, if Microsoft chose to develop
simultaneously for both OS/2 2.0 and Windows NT, the developers would encounter another set of problems. They
would have to add an abstraction layer to hide the differences in the operating systems, or they'd need substantial
reengineering for both, or they'd simply take a lowest-common-denominator approach and not fully use the services
or features of either system.

So Microsoft developers decided to stop work on the 32-bit version of SQL Server for OS/2 2.0. Instead, they
moved full-speed ahead in developing SQL Server for Windows NT, an operating system with an installed base of
zero. They didn't constrain the architecture to achieve portability back to OS/2 or to any other operating system. They
vigorously consumed whatever interfaces and services Windows NT exposed. Windows NT was the only horse, and
the development team rode as hard as they could. Except for bug fixing and maintenance, development ceased for
SQL Server for OS/2.

Microsoft began to inform customers that future versions, or a 32-bit version, of SQL Server for OS/2 2.0 would
depend on the volume of customer demand and that the primary focus was now on Windows NT. Most customers
seemed to understand and accept this position, but for customers who had committed their businesses to OS/2, this
was understandably not the message that they wanted to hear.
This document is created with the unregistered version of CHM2PDF Pilot

At this time, Sybase was working on a new version of its product, to be named System 10. As was the case when
they were working on version 4.2, the developers needed Microsoft SQL Server to be compatible with and have the
same version number as the Sybase release on UNIX. OS/2 vs. Windows NT was foremost at Microsoft, but at
Sybase, the success of System 10 was foremost.

Although System 10 wasn't even in beta yet, a schedule mismatch existed between the goal of getting a version of
Microsoft SQL Server onto Windows NT as soon as possible and getting a version of System 10 onto Windows
NT, OS/2 2.0, or both as soon as possible. The development team decided to compromise and specialize. Microsoft
would port SQL Server version 4.2 for OS/2 to Windows NT, beginning immediately. Sybase would bring Windows
NT under its umbrella of core operating systems for System 10. Windows NT would be among the first operating
system platforms on which System 10 would be available. In addition, Microsoft would turn the OS/2 product back
over to Sybase so those customers who wanted to stay with OS/2 could do so. Although Microsoft hoped to migrate
most of the installed customer base to Windows NT, they knew that this could never be 100 percent. They were
honestly pleased to offer those customers a way to continue with their OS/2 plans, via Sybase. The SQL Server
development team truly didn't want them to feel abandoned.

This compromise and specialization of development made a lot of sense for both companies. The development team
at Microsoft would be working from the stable and mature version 4.2 source code that they had become experts in.
Porting the product to a brand new operating system was difficult enough, even without having to worry about the
infant System 10 code base. And Sybase could concentrate on the new System 10 code base without worrying about
the inevitable problems of a prebeta operating system. Ultimately, System 10 and SQL Server for Windows NT
would both ship, and the two companies would again move back to joint development. That was the plan, and in
1992 both sides expected this to be the case.

The SQL Server team at Microsoft started racing at breakneck speed to build the first version of SQL Server for
Windows NT. Time to market was, of course, a prime consideration. Within Microsoft, the team had committed to
shipping within 90 days of the release of Windows NT; within the development group, they aimed for 30 days. But
time to market wasn't the only, or even chief, goal. They wanted to build the best database server for Windows NT
that they could. Because Windows NT was now SQL Server's only platform, the developers didn't need to be
concerned about portability issues, and they didn't need to create an abstraction layer that would make all operating
systems look alike. All they had to worry about was doing the best job possible with Windows NT. Windows NT
was designed to be a portable operating system, and it would be available for many different machine architectures.
SQL Server's portability layer would be Windows NT itself.

The development team tied SQL Server into management facilities provided by Windows NT, such as raising events
to a common location, installing SQL Server as a Windows NT service, and exporting performance statistics to the
Windows NT Performance Monitor. Because Windows NT allows applications to dynamically load code (using
DLLs), an open interface was provided to allow SQL Server to be extended by developers who wanted to write
their own DLLs.

This first version of Microsoft SQL Server for Windows NT was far more than a port of the 4.2 product for OS/2.
Microsoft essentially rewrote the kernel of SQL Server—the portion of the product that interacts with the operating
system—directly to the Win32 API.

Another goal of SQL Server for Windows NT was to make it easy to migrate current installations on OS/2 to the
new version and operating system. The developers wanted all applications that were written to SQL Server version
4.2 for OS/2 to work unmodified against SQL Server for Windows NT. Because Windows NT could be
dual-booted with MS-DOS or OS/2, the team decided that SQL Server for Windows NT would directly read from
and write to a database created for the OS/2 version. During an evaluation phase, for instance, a customer could
work with the OS/2 version, reboot the same machine, work with the Windows NT version, and then go back to
OS/2. Although it would be difficult to achieve, the developers wanted 100 percent compatibility.

The developers reworked SQL Server's internals and added many new management, networking, and extensibility
features; however, they didn't add new external core database engine features that would compromise compatibility.
This document is created with the unregistered version of CHM2PDF Pilot

The Windows NT-Only Strategy

Many have questioned the Windows NT-only strategy. But in 1992, the UNIX DBMS market was already
overcrowded, so Microsoft developers felt they wouldn't bring anything unique to that market. They recognized that
SQL Server couldn't even be in the running when UNIX was a customer's only solution. Microsoft's strategy was
based on doing the best possible job for Windows NT, being the best product on Windows NT, and helping to make
Windows NT compelling to customers.

The decision to concentrate on only Windows NT has had far-reaching effects. To be portable to many operating
systems, the Sybase code base had to take on or duplicate many operating system services. For example, because
threads either didn't exist on many UNIX operating systems or the thread packages differed substantially, Sybase had
essentially written its own thread package into SQL Server code. The Windows NT scheduling services, however,
were all based on a thread as the schedulable unit. If multiprocessors were available to the system and an application
had multiple runnable threads, the application automatically became a multiprocessor application. So the Microsoft
SQL Server developers decided to use native Windows NT threads and not the Sybase threading engine.

The development team made similar choices for the use of asynchronous I/O, memory management, network
protocol support, user authentication, and exception handling. (In Chapter 3, we'll look at the SQL Server
architecture in depth and delve into more of these specifics. The point here is that the goals intrinsic to portability are
in conflict with the goal to create the best possible implementation for a single operating system.)

For example, the developers ensured that there were no differences in the SQL dialect or capabilities of the Windows
NT and OS/2 versions. The plan was for the future System 10 version to implement many new features. This release
would be fundamentally a platform release. To emphasize compatibility with the OS/2-based 4.2 product and with the
Sybase product line, Microsoft decided to call the first version of SQL Server for Windows NT version 4.2. The
product was referred to as simply Microsoft SQL Server for Windows NT and often internally as SQL NT, a
designation that the press and many customers also were beginning to use.

In July 1992, Microsoft hosted a Windows NT developers' conference and distributed a prebeta version of
Windows NT to attendees. Even though SQL Server didn't exist yet at a beta level, Microsoft immediately made
available via CompuServe the 32-bit programming libraries that developers needed for porting their applications from
OS/2 or 16-bit Windows to Windows NT. Just as it had enjoyed success back in 1990 by providing the NDK to
prospective Windows 3.0 developers, Microsoft sought the same success by providing all the necessary tools to
would-be Windows NT developers.

Commitment to Customers

Incidentally, at approximately the same time they were delivering the NDK to developers, Microsoft also shipped a
maintenance release of SQL Server for OS/2 (and they shipped another the following year). In porting to Windows
NT, the development team found and fixed many bugs that were generic to all platforms. Even though they wouldn't
create new SQL Server versions for OS/2, they really meant it when they said that they didn't want to abandon the
OS/2 customers.

By March 1993, Microsoft went a step further and made a public beta release; anyone (even competitors) could
obtain this prerelease product, SQL Server Client/Server Development Kit (CSDK), for a nominal charge that
essentially covered expenses. Microsoft set up a public support forum on CompuServe and didn't demand that
participants sign a nondisclosure agreement. It shipped more than 3000 CSDKs. By May 1993, the volume on the
support forum for the prerelease product exceeded that for the shipping OS/2 product. The feedback was highly
positive: Microsoft had a winner. The dream of an eventual "client/server for the masses" was being realized.

In July 1993, Microsoft shipped Windows NT 3.1. Within 30 days, achieving their internal goal, the SQL Server
development team released the first version of Microsoft SQL Server for Windows NT to manufacturing. Customer
and press reaction was terrific. SQL Server for Windows NT was listed in many publications as among the top and
most important new products of 1993.
This document is created with the unregistered version of CHM2PDF Pilot

In October 1992, Microsoft shipped the first beta version of SQL Server for Windows NT. The product was
essentially feature-complete and provided a full Win32 version of all components. It shipped to more than 100 beta
sites. For a database server, having 100 beta sites was unprecedented; the typical number of beta sites for such a
product was approximately 10.
This document is created with the unregistered version of CHM2PDF Pilot

34

Success Brings Fundamental

Change
SQL Server for Windows NT was a success by nearly all measures. The strategy of integrating the product tightly
with Windows NT resulted in a product that was substantially easier to use than high-end database products had ever
been. Sales were above the internal projections and increased as Windows NT began to win acceptance.

By early December 1993, much of the SQL Server customer base for the OS/2 operating system had already
migrated to SQL Server for Windows NT. Surveys showed that most of those who hadn't yet upgraded to Windows
NT planned to do so, despite the fact that Sybase had publicly announced its intention to develop System 10 for
OS/2.

The upgrade from SQL Server for OS/2 to SQL Server for Windows NT was virtually painless. When applications
were moved from one platform to the other, not only did they still work, but they worked better. SQL Server for
Windows NT was much faster than SQL Server for OS/2, and most significant, it was scalable far beyond the limits
imposed by OS/2. Within another nine months, Microsoft's SQL Server business had more than doubled, with nearly
all sales coming on the Windows NT platform. Although they continued to offer the OS/2 product, it accounted for
well below 10 percent of sales.

A Faster SQL Server

Focusing the product on Windows NT had made the product fast. Studies conducted internally at Microsoft, as well
as those based on private customer benchmarks, all showed similar results: SQL Server for Windows NT (running on
low-cost commodity hardware) was competitive in performance with database systems running on UNIX (much
more costly hardware).

In September 1993, Compaq Computer Corporation published the first official, audited Transaction Processing
Council (TPC) benchmarks. At that time, on the TPC-B benchmark, well over $1000/TPS (transactions per second)
was common, and it was an impressive number that broke the $1000/TPS barrier. Running on a dual-Pentium,
66-MHz machine, SQL Server achieved 226 TPS at a cost of $440 per transaction, less than half the cost of the
lowest benchmark ever published by any other company. The raw performance number of 226 TPS was equally
astonishing. At that time, most of the TPC-B numbers on file for UNIX minicomputer systems were still below 100
TPS. Certainly a handful of numbers were higher, but all occurred at a price point of much more than $440/TPS. Just
18 months prior, the raw performance of 226 TPS was about as high as any mainframe or minicomputer system had
ever achieved.

The implications were clear. SQL Server for Windows NT wasn't simply a low-end or workgroup system. Its
performance was competitive with more costly UNIX systems, and the trend toward faster systems running Windows
NT was unmistakable. The 66-MHz Pentium processors were the first generation of Pentiums from Intel. Much
higher clock speeds were expected to emerge within a few months; hardware with additional processors was
anticipated. Furthermore, Microsoft SQL Server for Windows NT would soon be available on RISC processors
such as the DEC Alpha-AXP at 250 MHz and the MIPS R4400. The so-called Moore's Law (named after Gordon
Moore, cofounder of Intel Corporation), which postulates that computing power doubles every 18 months, was
clearly being proven true for the type of commodity hardware for which Windows NT and Microsoft SQL Server
were designed. Microsoft took a serious look at what would be required to achieve 1000 TPS, definitely putting raw
performance in the same league as even the largest systems of the day.
This document is created with the unregistered version of CHM2PDF Pilot

34

The End of Joint Development

Microsoft's success strained its relationship with Sybase. The competitive landscape of late 1993 was quite different
from that of 1987, when Microsoft and Sybase had inked their deal. By 1993, Sybase was a successful software
company, by most accounts second only to Oracle in the DBMS market. The credibility and visibility Microsoft
brought to Sybase was far less important in 1993 than it had been to the upstart company of 1987. Similarly,
Microsoft had grown a great deal since 1987. The growth wasn't just in revenues (although that was one of the great
success stories of the industry) but also in the development of products for use with enterprise applications, such as
Microsoft SQL Server, that Fortune 1000 companies could use as a platform on which to run their businesses.

The SQL Server development team had grown as well, from a handful of people in 1990 to more than 50
professionals (not including those in marketing, support, or field operations), with significant additional growth
planned. The first-rate team of engineers knew database and transaction processing and the inner workings of SQL
Server, and they were experts in developing for Windows NT. Microsoft now had the talent, size, motivation, and
mandate, but it was still constrained in what it could do with the product: the 1987 agreement with Sybase had merely
licensed to Microsoft the rights to the Sybase product. Because of this restricted agreement, Microsoft couldn't
implement new features or changes without Sybase's approval. Contrary to what many people thought, Microsoft had
no ownership stake in Sybase and could by no means simply call the shots.

Obviously, Sybase had different business needs and therefore different priorities than Microsoft. The development
team at Microsoft might, for example, want to integrate SQL Server with messaging by using MAPI (Messaging
API), but because this feature was specific to the Microsoft operating system, Sybase wouldn't be excited about it.
As is always the case in development, many features could be implemented for every feature that is actually
implemented: features specific to Windows NT didn't tend to interest Sybase as much as those that would benefit its
UNIX products.

Sybase engineers had to confront the issue of portability to multiple operating systems. In fact, Microsoft's
implementation of version 4.2 for Windows NT was already causing friction because Sybase was progressing with
System 10 for Windows NT. Sybase was understandably implementing System 10 in a more portable manner than
Microsoft had done. This was entirely rational for Sybase's objectives, but from Microsoft's perspective, it meant a
looser bond with Windows NT. System 10 would not, and could not, perform as well on Windows NT as the
product that had been designed and written exclusively for Windows NT.

Because of the economics involved, as well as the changing competitive landscape, the Microsoft/Sybase agreement
of 1987 was no longer working. Microsoft SQL Server was now a viable competitive alternative to Sybase SQL
Server running on UNIX, Novell NetWare, and VMS. Far from seeding the market for Sybase, Microsoft SQL
Server was now taking sales away from Sybase. Instead of choosing a UNIX solution, customers could buy
Microsoft SQL Server at a fraction of the cost of a UNIX solution, run it on less expensive PC hardware, and install
and administer it more easily. Although Sybase earned royalties on sales of Microsoft SQL Server, this amounted to a
small fraction of the revenue Sybase would have received if customers had bought Sybase products for UNIX in the
first place. Microsoft and Sybase were now often vigorously competing for the same customers. Both companies
recognized that a fundamental change in the relationship was needed.

On April 12, 1994, Microsoft and Sybase announced an end to joint development. Each company decided to
separately develop its own SQL Server products. Microsoft was free to evolve and change Microsoft SQL Server.
Sybase decided to bring its System 10 products (and subsequent versions) to Windows NT—the first time that the
Sybase-labeled SQL Server would be available for a Microsoft operating system. (The original agreement gave
Microsoft exclusive rights on Microsoft operating systems.) Both companies' products would be
backward-compatible with existing SQL Server applications—however, the products would diverge in the future and
would have different feature sets and design points. Sybase's product would be fully compatible with its UNIX
This document is created with the unregistered version of CHM2PDF Pilot

versions. Microsoft's product would continue to be integrated with Windows NT as much possible. In short, the
products would directly compete.
SQL Server Performance: No Secret Formulas

Although Microsoft SQL Server is designed for and optimized for Windows NT, it uses only publicly documented
interfaces. From time to time, articles or speakers suggest that Microsoft SQL Server uses private, undocumented
APIs in Windows NT to achieve its performance. But this assumption is false, without exception. SQL Server's
performance results from using the available and published Windows NT services without any compromise. This is
something that other products could also do if developers were willing to dedicate themselves to doing the best
possible job for Windows NT without making compromises for portability to other operating systems.
NOTE
Although the Sybase product competed with Microsoft SQL Server, Microsoft encouraged and provided support
for getting Sybase System 10 shipping on Windows NT as soon as possible because it was important to the
acceptance and success of Windows NT. Such collaboration is typical; many such relationships are competitive on
one level and cooperative on another.
This document is created with the unregistered version of CHM2PDF Pilot

34

The Charge to SQL95

As late as the beginning of 1994, the SQL Server development team had planned for the next version of the product
to pick up the Sybase System 10 source code and new features. But the termination of joint development changed
that plan. Sybase contributed no more code, and Microsoft released no System 10 product. Except for a couple of
bug fixes, the last code drop received from Sybase was in early 1992 just as version 4.2 for OS/2 was shipping.

Time was at a premium. Besides promoting sales to new customers, Microsoft had to fight for its significant installed
base. Sybase was to deliver System 10 for Windows NT later that year. That would be a potentially easy upgrade for
Microsoft's installed customer base; so if Microsoft lost customers to System 10, they'd likely lose them forever.

Microsoft quickly planned for an ambitious release that was loaded with new features and performance
improvements. It was tagged SQL95, borrowing the working moniker from the well-known upcoming Windows 95
release. Because the Big Question of 1994 was "What are your plans for replication?", replication became a keystone
of the release. So did scrollable cursors, a feature that the developers had learned was necessary (from the ill-fated
dBASE IV interface) to bridge the impedance mismatch between many record-oriented applications and a relational
database. No mainstream DBMS product had yet provided a fully functional implementation of scrollable cursors in a
client/server environment, and the SQL Server team believed that it was imperative to add this feature to the database
engine. They had also been working on a dramatic new set of management tools, code-named Starfighter (today's
SQL Server Enterprise Manager), which would also be included in the next release. The new feature list went on and
on.

Microsoft's customers were clamoring to hear about the plans for SQL Server post-Sybase. So on June 14, 1994,
Microsoft put together a briefing event in San Francisco for key customers, analysts, and the press. Jim Allchin,
Microsoft senior vice president, walked the attendees through the plans for the future and for the SQL95 release.
Attendees were impressed with the plans and direction, but many were openly skeptical of Microsoft's ability to
deliver such an impressive product by the end of 1995.
NOTE
Some industry press began to sarcastically refer to the planned release as SQL97 and even SQL2000. Internally, the
development team was still targeting the first half of 1995. To outsiders, they were more cautious—after all, this is
software development, which is still more art than science. Stuff happens, so the SQL Server team said nothing more
ambitious than 1995. (Everyone assumed that the planned date was December 31, 1995, and that they'd miss that
date, too.) The rampant skepticism only served to motivate the SQL Server team even more to show that they could
deliver. After all, the team rightly felt that it had already delivered an impressive release independent of Sybase. But
no one gave them credit for that, so they'd just have to show everyone.

The team worked incredibly hard, even by Microsoft standards, to make this deadline and still deliver a full-featured,
high-quality product. The first beta was released at the end of October 1994. Although Starfighter was not
feature-complete yet, the database server was complete, and because the server takes the longest lead time for beta
sites to really stress it, they went ahead with the beta release. This release was followed by a series of beta updates
during the next several months, along with gradual beta-site expansion, eventually surpassing 2000 sites.

For nine months, dinner was delivered each night for those on the development team who were working late—usually
a majority. On June 14, 1995, the product was released to manufacturing. Microsoft SQL Server 6.0 (SQL95) had
shipped within the original internal target date, much sooner than nearly everyone outside the team had expected. It
was an immediate hit. Positive reviews appeared in nearly all the trade publications; even more significant, none were
negative or even neutral. Even more important than the press reviews was customer reaction, which was terrific.

InfoWorld, in its second annual survey of the 100 companies with the most innovative client/server applications in the
This document is created with the unregistered version of CHM2PDF Pilot

previous year, showed Microsoft SQL Server as the number-two database. SQL Server jumped from 15 percent to
18 percent of those surveyed as the database server of choice—for a virtual tie with Oracle, which dropped from 24
percent to 19 percent. Sybase rose from 12 to 14 percent. Three of the top 10 applications highlighted by InfoWorld
were built using Microsoft SQL Server.

Of course, the SQL Server development team was happy to see this data, but they took it with a grain of salt. Other
data could be interpreted to suggest that Microsoft SQL Server's presence was substantially less than the surveys
indicated. And the team recognized that they were still relative newcomers. From a sales perspective, Oracle was
clearly king of the hill, and Sybase, Informix, and IBM were also formidable competitors in the DBMS market.
Microsoft had not previously been a significant competitive threat. But now it was, and all companies armed their
sales forces with bundles of information on tactics for selling against Microsoft SQL Server. Sybase, Informix, and
Oracle all promised hot new releases. Although the SQL Server team had been sprinting for nearly four years, now
was certainly not the time to get complacent.
Team-Building with Top Talent

Besides working hard on the development of version 6.0, Microsoft was also working to increase the size and
strength of the team. It had built a small, crackerjack team that had delivered SQL Server for Windows NT, and this
team was the core that delivered SQL95. But the team needed more people, more expertise, and exposure to
broader ideas. So they went after top talent in the industry.

Microsoft attracted some industry luminaries—Jim Gray, Dave Lomet, and Phil Bernstein. They also attracted a lot
of lesser-known but top development talent from throughout the industry. For example, DEC shopped its Rdb
product around, looking to generate cash, and Oracle eventually spent a few hundred million dollars for it. But
Microsoft didn't want to buy Rdb. Instead, it hired many of the Rdb project's best developers, augmenting this move
by hiring several of the best recent masters' graduates who had specialized in databases.
This document is created with the unregistered version of CHM2PDF Pilot

34

The Next Version

After shipping version 6.0, many team members took well-deserved vacations after months of working in crunch
mode. But within a month, they were actively beginning work on SQL Server version 6.5. With any huge release like
version 6.0, some features get deferred due to schedule constraints. And during the ambitious 18-month project, new
demands had come up that weren't even conceived of as requirements when the project began. For example, the
Internet and data warehousing both exploded in importance and demand in 1995. Version 6.5 added capabilities for
both. It also included further ease-of-use improvements, gained certification for conforming to the ANSI SQL
standard, and provided much richer distributed transactions.

Although version 6.0 was released to manufacturing in June 1995, on December 15, 1995, Microsoft shipped a
feature-complete beta version of 6.5 to 150 beta sites. The production version of 6.5 was released to manufacturing
in April 1996, a scant 10 months after 6.0 was released. The SQL Server team was not about to slow down.
This document is created with the unregistered version of CHM2PDF Pilot

34

The Secret of the Sphinx

Slowing down was never really an option. Even before SQL Server 6.5 was released, an entirely separate team of
developers was hard at work on a brand new vision for Microsoft SQL Server. In 1993, Microsoft had decided that
databases would be a core technology in the product line, and in late 1994, it began hiring expertise from DEC and
other major vendors to work with Microsoft's Jet and SQL Server teams to plan components for a whole new
generation of database technology. During 1995, the period when SQL Server 6.0 and SQL Server 6.5 were being
released, this team was building a new query processor that was planned as a component for what was to become the
Microsoft Data Engine (MSDE).

Development of the MSDE was enhanced by the simultaneous development of OLE DB, which allowed elements of
the SQL Server core product to be developed as independent components. Components could then communicate
with each other using an OLE DB layer. At the end of 1995, the new query-processing component was integrated
into the SQL Server code base, and development of a brand new SQL Server, with the code name Sphinx, began in
earnest. The team working on the query processor joined the rest of the SQL Server development team.

The development of this new generation of SQL Server had one overriding theme: to re-architect the entire database
engine so that the SQL Server product could scale as much as its users wanted it to. This would mean upward growth
to take full advantage of more and faster processors and as much memory as the operating system could handle. This
growth would also have to allow for new functionality to be added to any of the components so that, for example, the
query processor code could add an entirely new join algorithm to its repertoire as easily as a new hard drive could be
added. In addition to the upward growth, SQL Server would also be expanded outward to support whole new
classes of database applications. It would also need to scale downward to run on smaller systems such as desktops
and laptops.

There were two immediate goals for the first version of this new re-architected system:
 Full row-level locking with an extremely smart lock manager.
 A brand-new query processor that would allow such techniques as distributed heterogeneous queries and
efficient processing of ad hoc queries. (Ad hoc queries are a fact of life when you deal with data warehousing,
Internet-based applications, or both.)

This brand-new SQL Server version 7.0 debuted in a limited beta 1 release in the fall of 1997. Beta 2 was made
available to several hundred sites in December 1997. Because of the new architecture, all databases and the data
structures they contained needed to be completely rebuilt during the upgrade process. Microsoft and the SQL Server
development team were absolutely committed to making sure all customers would be successful in moving from SQL
Server 6.5 to SQL Server 7.0. A program called the 1K Challenge was instituted, in which 1000 customers were
invited to send copies of their databases to the SQL Server development team to be upgraded to version 7.0. A
porting lab was set up in the building on the Redmond campus where the entire development team worked. Each
week from February to August of 1998, four or five ISVs sent a team of their own developers to Microsoft to spend
an entire week in the lab, making sure their products would work out of the box with the new SQL Server 7.0. The
core SQL Server development engineers made themselves available to immediately isolate and fix any bugs
encountered and to spend time talking to the ISVs about the best ways to take advantage of all the new features of
SQL Server 7.0 to make their own products even better.

In June 1998, Microsoft publicly released SQL Server 7.0 Beta 3 from its SQL Server Web site, along with a set of
exercises demonstrating many of the new features and capabilities of the product. An Internet news server was set up
so that any Beta 3 user could report bugs and ask questions of the SQL Server development team. More than
120,000 sites received the Beta 3 version of SQL Server 7.0. This included companies directly requesting the
product through the Microsoft Web site, MSDN subscriptions, and Microsoft Beta Program participants (who
receive beta versions of all Microsoft products when they become available). And then at last, after almost four years
This document is created with the unregistered version of CHM2PDF Pilot

for some of the earliest team members, SQL Server 7.0 was publicly announced at COMDEX in Las Vegas on
November 16, 1998. By the time of the launch, more than a dozen sites were in full production with SQL Server 7.0,
including Microsoft's internal SAP implementation and barnesandnoble.com, HarperCollins, CBS Sportsline,
Comcast Cellular, and Southwest Securities. The SQL Server 7.0 development team released the product to
manufacturing on December 2, 1998, using build 7.00.623.07, with a code freeze date of November 27, 1998. The
product became available to the public in January 1999.
This document is created with the unregistered version of CHM2PDF Pilot

34

Software for the New Century

As expected, the development team didn't stop working on SQL Server. Several planned features that had not made
it into the SQL Server 7.0 release needed to be incorporated into the product. Other new features were already in the
later stages of development in preparation for a version after 7.0. Two new code names were already in use: Shiloh
was to be the version after 7.0, commonly assumed to be just a "dot" release like 7.5, and Yukon was the name given
to the next major revision of SQL Server.

Initially, the SQL Server product managers were reluctant to predict widespread, immediate acceptance of SQL
Server 7.0. Because it was basically a complete rewrite of the core engine code, many people might consider it a
1.0-level release. It was also anticipated that many potential adopters and upgraders would not be anxious to go to
any release that ended in dot-zero and would want to wait for at least the first service pack. So initially, the Shiloh
release was planned as just a "Super Service Pack." It would include perhaps a few new features that didn't get into
7.0 because of the shipping schedule, plus any bug fixes that were deemed too big for a normal, unnumbered service
pack. The plan was to have a quick turnaround and release Shiloh no more than a year after the initial release of SQL
Server 7.0.

Several factors came into play to change this initial vision of Shiloh. First, there seemed to be very little hesitation
among the already installed user base to upgrade to the new SQL Server 7.0. Also, sales figures for new customers
far exceeded even the most optimistic predictions. People were using SQL Server 7.0, and they liked it. Even after
the public release of the product, Microsoft continued running the ISV lab in the building where the SQL Server
developers worked, so feedback was ongoing and readily available to the development team. In general, the issues
that came up as problems were easily fixable and a service pack for SQL Server 7.0 was released in May 1999. A
second service pack was released in March 2000. There seemed to be no need for the "Super Service Pack" that
Shiloh was intended to be.

A second factor that influenced a change in vision for the Shiloh release was customer requests. After referential
integrity with cascading updates and deletes failed to make it into SQL Server 7.0, the product team heard loud and
clear from the customer base that any further delay in adding this feature would not be tolerable. Customers also
wanted better support for partitioned views and optimizer support for the star schema design used in data
warehousing applications.

Finally, there was competitive pressure to make the next release bigger and better than originally planned. The Million
Dollar Challenge issued by Larry Ellison of the Oracle Corporation pointed out a major piece of functionality that
"they" had and "we" didn't. Adding materialized views to the product, which would allow SQL Server to meet this
challenge, would be more than just a simple fix.

So the decision was made to have Shiloh be a full-blown release with at least an 18-month development cycle, but
the version number would still be 7.5. The magnitude of the changes that would eventually find their way into this
release was not immediately clear because the only absolute "must-have" at the outset was the cascading updates and
deletes. It soon became clear that the release would grow beyond initial expectations. The team itself had grown
substantially, spilling out of their original building on the main Microsoft campus into parts of two adjacent buildings.
With a larger group of developers, a greater number of small to medium-size new features could be added to the
product without seriously jeopardizing the planned release date. This book describes many of these new features.

In addition to having to add a substantial amount of new functionality, the development team also imposed upon
themselves something they called "stretch goals." They declared an objective of a 20 percent increase in performance
across the board, on all types of applications, but to give themselves something concrete to aim for, they extended this
for specific applications. The main example of this was their goal to improve performance on the SAP R/3 Sales and
Distribution benchmark by at least 40 percent. To do this, they had to make very specific changes in the optimizer that
This document is created with the unregistered version of CHM2PDF Pilot

would directly affect the SAP queries but also benefit many other applications. At the Windows 2000 launch event in
San Francisco on February 17, 2000, benchmark results were announced that demonstrated 6700 users for the Sales
and Distribution benchmark, compared to 4500 users in SQL Server 7.0 for the same benchmark on the same
hardware (an eight-way Pentium III-550). This represented a 48 percent improvement, and the goal had been well
met.

After the development period was extended to a full 18 months, the decision was made to add another new feature.
This decision was held in the strictest secrecy and wasn't even discussed with many high-level executives at Microsoft.
The feature wasn't even addressed until after the release of Beta 1 in November 1999, and it was not announced
publicly until the February launch of Windows 2000. Referred to internally by the code name Coyote, this secret
project allowed SQL Server 2000 to support distributed partitioned views to allow unprecedented scalability in
data management operations. It was this new feature that allowed the world-record-breaking benchmark results to be
achieved and announced in San Francisco in February 2000.

Originally, these scalability features were scheduled for the version after Shiloh, but a close investigation revealed that
many of the necessary components were already in place. These features included extensions to the optimizations of
union views and the ability to update union views. Details of distributed partitioned views will be discussed later in the
book.

The initial Beta 1 of Shiloh was released to a very small group of Early Adopters and serious beta testers in
September 1999. Shortly after that, Microsoft announced that the official name of the product would be SQL Server
2000. There were two main reasons for this change. First, because of everything that was changing about the product,
it was unrealistic to make it just a dot release (7.5); it really needed a whole new number. Second, if the product had
gone with the name 8.0, it would be the only Microsoft BackOffice product that didn't use the name 2000. To
conform to this companywide naming standard, the name SQL Server 2000 was announced. However, if you look at
the internal version number using the @@VERSION function, you'll see that it returns the number 8.00.194.

It turns out that from the user's perspective, SQL Server 2000 introduces even more new functionality than did its
immediate predecessor. SQL Server 7 had a completely rewritten engine, including brand-new storage structures,
data access methods, locking technology, recovery algorithms, transaction log architecture, memory architecture, and
optimizer. These, of course, are the things that this book describes in detail. But for the end user, developer, or
part-time database administrator (DBA), the external changes and language enhancements in SQL Server 7 were
minimal. SQL Server 2000 adds dozens of new language features as well as drastic changes to existing objects such
as table constraints, views, and triggers that all developers and most DBAs will need to know about.

Because the internal engine changes are minimal, only two betas were planned. Beta 2, released in April 2000, was
the widespread public beta and was sent to thousands of interested users, conference attendees, ISVs, and
consultants. The SQL Server 2000 development team froze the code at build 8.00.194.01, on August 6, 2000, and
released the product to manufacturing on August 9, 2000.

Internal SQL Server development is a never-ending story. Just as after the release of SQL Server 7, new capabilities
were already well past the design stage, ready to be implemented. Features that didn't make it into SQL Server 2000,
as well as continued improvements in scalability, availability, interoperability, and overall performance, continue to be
addressed. The development team is using feedback from the beta forums and from early users of the released
version of SQL Server 2000 to shape the future of Microsoft SQL Server.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 2

A Tour of SQL Server

Microsoft SQL Server 2000 is a high-performance client/server relational database management system (RDBMS).
It was designed to support high-volume transaction processing (such as that for online order entry, inventory,
accounting, or manufacturing) as well as data warehousing and decision-support applications (such as sales analysis
applications). SQL Server 2000 runs on Microsoft Windows NT 4 or Microsoft Windows 2000 Server-based
networks using Intel processors an d can be installed as a personal desktop database system on machines running
Windows NT Workstation 4, Windows 2000 Professional, Windows 98, and Windows Millennium Edition (Me).
You can use the same CD to install one of the Server versions or the Personal version of SQL Server 2000. In
addition, you can install multiple instances of SQL Server 2000 on the same computer, each with its own set of users
and its own data. (I'll cover the details of multiple instances in Chapter 4.)

SQL Server 2000 actually has seven different editions available on different CDs: Standard Edition, Enterprise
Edition, Personal Edition, Developer Edition, Windows CE Edition, Evaluation Edition, and Microsoft Desktop
Engine (MSDE). (I'll discuss the differences between the editions in Chapter 4.)
NOTE
To simplify operating system designations, I'll use the following conventions. Windows NT/2000 means any of the
following operating systems that support SQL Server 2000: Windows NT Server 4, Windows NT Server 4,
Enterprise Edition, Windows NT Workstation 4, Windows 2000 Server, Windows 2000 Advanced Server,
Windows 2000 Professional, or Windows 2000 Datacenter Server. If I need to refer specifically to just the server
platforms, I'll use the designation Windows NT/2000 Server, which means any of the operating systems listed above
except Windows NT Workstation 4 and Windows 2000 Professional. To refer to just the non-server operating
systems, I'll use Windows NT/2000 Workstation. The designation Windows 98 includes both Windows 98 and
Windows Me.

SQL Server 2000 also provides many client tools and networking interfaces for other Microsoft operating systems,
such as Windows 3.1 and MS-DOS. And because of SQL Server's open architecture, other systems (such as
UNIX-based systems) can also interoperate with it. SQL Server is part of the core of a family of integrated products
that includes development tools, systems management tools, distributed system components, and open development
interfaces, as shown in Figure 2-1. It's also a key part of Microsoft BackOffice.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 2-1. The SQL Server family of integrated components.

Most of this book focuses on the capabilities and uses of the SQL Server engine; this chapter provides an overview
of the SQL Server family of components and describes the features and benefits of each component. Understanding
these features and benefits will be helpful to you as you develop applications.
This document is created with the unregistered version of CHM2PDF Pilot

34

The SQL Server Engine

The SQL Server engine supports a variety of demanding applications, such as online transaction processing (OLTP)
and decision-support applications. At the core of its decision-support capabilities is Transact-SQL, Microsoft's
version of Structured Query Language (SQL). Beneath this query language are the components that support
transaction processing and recoverability.

Transact-SQL
SQL is a well-known, widely used data access language. Every mainstream database management system (DBMS)
product implements SQL in some way. Transact-SQL (often referred to as T-SQL) is a powerful and unique superset
of the SQL standard.

The SQL SELECT statement provides tremendous power and flexibility for retrieving information. Using SELECT,
data from multiple tables can be easily chosen and combined and the results returned in tabular format.

Take a look at the following two tables from the pubs sample database. (This database, which I use in many
examples in this book, is installed when SQL Server is installed. For brevity, I'll sometimes use an abbreviated amount
of the data, as in this example.)
publishers Table
pub_id pub_name city state
0736 New Moon Books Boston MA
0877 Binnet & Hardley Washington DC
1389 Algodata Infosystems Berkeley CA

titles Table
title_id title pub_id
BU1032 The Busy Executive's Database Guide 1389
BU1111 Cooking with Computers: Surreptitious 1389
Balance Sheets
BU2075 You Can Combat Computer Stress! 0736
BU7832 Straight Talk About Computers 1389
MC2222 Silicon Valley Gastronomic Treats 0877
MC3021 The Gourmet Microwave 0877
MC3026 The Psychology of Computer Cooking 0877

The following simple SELECT statement logically joins the titles and publishers tables to retrieve the book titles with
their corresponding publishers:
SELECT title, pub_name, city, state
FROM titles JOIN publishers
ON titles.pub_id = publishers.pub_id

Here's the result:

title pub_name city state
----- -------- ---- -----
The Busy Executive's Database Algodata Infosystems Berkeley CA
Guide
Cooking with Computers: Algodata Infosystems Berkeley CA
This document is created with the unregistered version of CHM2PDF Pilot

Surreptitious Balance Sheets

You Can Combat Computer Stress! New Moon Books Boston MA
Straight Talk About Computers Algodata Infosystems Berkeley CA
Silicon Valley Gastronomic Binnet & Hardley Washington DC
Treats
The Gourmet Microwave Binnet & Hardley Washington DC
The Psychology of Computer Binnet & Hardley Washington DC
Cooking

The preceding query, a simple SQL statement, shows that standard SQL provides a powerful way to query and
manipulate data. (In Chapters 7 and 10, I'll discuss SQL queries in much greater depth.)

The Transact-SQL language is compliant with the American National Standards Institute (ANSI) SQL-92 standard
at the entry level. It offers considerably more power because of its unique extensions to the standard.

Transact-SQL extensions
Transact-SQL provides a number of capabilities that extend beyond typical implementations of SQL. These
capabilities allow you to easily and efficiently write queries that are difficult to write in standard SQL. You can, for
example, embed additional SELECT statements in the SELECT list, and you can drill into a result set by further
selecting data directly from a SELECT statement, a feature known as a derived table. Transact-SQL provides many
system functions for dealing with strings (for finding substrings and so on), for converting datatypes, and for
manipulating and formatting date information.

Transact-SQL also provides mathematical operations such as returning the square root of a number. In addition, you
can use special operators such as CUBE and ROLLUP to efficiently perform multidimensional analysis at the
database server, where the analysis can be optimized as part of the execution plan of a query. You can use the CASE
expression to easily make complex conditional substitutions in the SELECT statement. Multidimensional (sometimes
referred to as OLAP, or online analytical processing) operators, such as CUBE, and conditional expressions, such as
CASE, are especially useful in implementing data warehousing solutions with SQL Server.

The query optimizer

In Transact-SQL, a cost-based query optimizer determines the likely best way to access data. This automatic
optimization allows you to concentrate on defining your query criteria rather than defining how the query should be
executed. For example, this nonprocedural approach eliminates the need for you to know which indexes exist and
which, if any, should be used. Would it be more efficient to incur additional I/Os to read index pages in addition to
data pages, or would it be better just to scan the data and then sort it? The optimizer automatically, invisibly, and
efficiently resolves these types of important questions for you.

The query optimizer maintains statistics about the volume and dispersion of data, which it then uses to determine the
execution plan that will most likely work best for the operation requested. Because a cost-based optimizer is by
definition probability-based, an application might want to override the query optimizer in some special cases. In your
application, you can specify optimizer hints that direct the choice of execution plan. In addition, you can use one of
SQL Server's SHOWPLAN options to explain the chosen execution plan and provide insight into why it was
chosen—this is information that allows you to fine-tune the application and database design.

The programmable server

Transact-SQL provides programming constructs—such as variables, conditional operations (IF-THEN-ELSE), and
looping—that can dramatically simplify application development by allowing you to use a simple SQL script rather
than a third-generation programming language (3GL). These branching and looping constructs can dramatically
improve performance in a client/server environment by eliminating the need for network conversations. Minimizing
network latency is an important aspect of maximizing client/server application performance. For example, instead of
This document is created with the unregistered version of CHM2PDF Pilot

having the server return a value that the client needs to evaluate in order to issue a subsequent request, you can build
conditional logic directly into the SQL batch file so that the routine is evaluated and executed entirely at the server.

You can use Transact-SQL to write complex batches of SQL statements. (A batch of SQL statements in a complex
application can potentially be hundreds, or even thousands, of lines long.) An important capability of SQL Server
2000 is the T-SQL Debugger, which allows you to use SQL Query Analyzer to fully debug Transact-SQL routines,
including stepping through the statements, setting breakpoints, and setting watchpoints on Transact-SQL variables.

Stored procedures and functions

Simply put, stored procedures and functions are collections of SQL statements stored in a SQL Server database.
You can write complex queries and transactions as stored procedures and then invoke them directly from the
front-end application. Whenever an ad hoc SQL command is sent to a database server for processing, the server
must parse the command, check its syntax for sense, determine whether the requester has the permissions necessary
to execute the command, and formulate an optimal execution plan to process the request. Stored procedures and
functions execute faster than batches of dynamic SQL statements, sometimes dramatically faster, because they
eliminate the need to reparse and reoptimize the requests each time they're executed. SQL Server supports stored
procedures that let you store groups of compiled SQL statements on the server for later recall, to limit the overhead
when the procedures are subsequently executed.

Stored procedures and functions differ from ad hoc SQL statements and from batches of SQL statements in that
they're checked for syntax and compiled only the first time they're executed. SQL Server stores this compiled
version in its memory cache and then uses the cached, compiled version to process subsequent calls, which results in
faster execution times. Stored procedures and functions can also accept parameters, so they can be used by multiple
applications using different input data.

Stored procedures and functions differ from each other in how they're used in your code. Stored procedures perform
actions and usually contain all the error checking associated with that action. Many developers prefer to have each
procedure perform a single, well-defined action. For example, a stored procedure might be written to insert a new
order into the orders table, and the data must be checked for validity before the actual insert occurs. A function, on
the other hand, returns a value, and that value can be used anywhere that any SQL statement can use a single value.
For example, you can use a function in the list of items to return in the SELECT clause, you can compare it to other
values in a WHERE clause, you can use it to generate a value for insertion in the VALUES clause of an INSERT
statement, or you can use it to supply a parameter to a stored procedure or to another function. In addition, functions
in SQL Server 2000 can return a table-valued result that you can use in a FROM clause. I'll discuss stored
procedures and functions in detail in Chapter 11.

You can think of stored procedures as basic building blocks in database application development. Even if stored
procedures provided no performance advantage (which, of course, they do), you'd still have a compelling reason to
use them: they provide an important layer of insulation from changes in business practices. Suppose, for example, that
a retailer maintains a mailing list for catalog distribution, which is used by multiple applications. After the applications
are deployed, a change in criteria and logic (that is, the business rules) occurs, which affects which customers should
automatically receive new catalogs. If the business rules were programmed directly into the company's applications,
every application would need to be modified—probably an expensive and time-consuming operation. Furthermore, if
multiple developers worked on the applications, the rules might not have been programmed with exactly the same
semantics by every programmer.

A stored procedure, on the other hand, can be modified once, in seconds, at the server. An application need not be
changed or even recompiled. The next time the application executes the stored procedure, the new rules are in place
automatically. In addition to providing a performance advantage, stored procedures can provide an important security
function. By granting users access to a stored procedure but not to the underlying tables, you can allow them to
access or manipulate data only in the way prescribed by the stored procedure.

Extended stored procedures

This document is created with the unregistered version of CHM2PDF Pilot

Extended stored procedures, which are unique to SQL Server, allow you to extend the programming capabilities
provided by Transact-SQL and to access resources outside of SQL Server. Messaging integration, security
integration, the ability to write Hypertext Markup Language (HTML) files (files formatted for use on the Internet), and
much of the power of SQL Enterprise Manager (discussed later in this chapter) are all implemented using extended
stored procedures. You can create extended stored procedures as external dynamic link libraries (DLLs). For
example, you can write a DLL to establish a modem connection, dial the ACME Credit Service, and return a status
value indicating credit approval or rejection. Also, you can build extended stored procedures using the Microsoft
Open Data Services (ODS) API, which lets your stored procedures return self-describing result sets to the calling
client applications, just as a normal stored procedure would.

Extended stored procedures allow even Microsoft to extend SQL Server. Good engineering practices dictate that if
code doesn't benefit from being shared, it should be segregated and isolated. With this principle in mind, Microsoft
added integration with messaging via the Messaging API (MAPI) as a set of extended stored procedures (
xp_sendmail, xp_readmail, and so on) instead of directly modifying the SQL Server engine. Extended stored
procedures allow you to add powerful features without risk of disrupting the core server engine. You can thus add
more features quickly, with less risk of destabilizing the server. And because an extended procedure's DLL is loaded
only if the procedure is executed, the memory footprint of SQL Server doesn't grow for services that aren't being
used.
This document is created with the unregistered version of CHM2PDF Pilot

34

DBMS-Enforced Data Integrity

A database is only as useful as the user's confidence in it. That's why the server must enforce data integrity rules and
business policies. SQL Server enforces data integrity within the database itself, guaranteeing that complex business
policies are followed and that mandatory relationships between data elements are complied with.

Because SQL Server's client/server architecture allows you to use a variety of front-end applications to manipulate
and present the same data from the server, it would be cumbersome to encode all the necessary integrity constraints,
security permissions, and business rules into each application. If business policies were all coded in the front-end
applications, every application would need to be modified every time a business policy changed. Even if you
attempted to encode business rules into every client application, the danger of an application misbehaving would still
exist. Most applications can't be fully trusted. Only the server can act as the final arbiter, and the server must not
provide a back door for a poorly written or malicious application to subvert its integrity.

SQL Server uses advanced data integrity features such as stored procedures, declarative referential integrity (DRI),
datatypes, constraints, rules, defaults, and triggers to enforce data integrity. Each of these features has its own use
within a database; by combining these integrity features, you can make your database flexible and easy to manage yet
secure.

Declarative Data Integrity

The ability to enforce data integrity based on the constraints you define when you create the database tables is known
as declarative data integrity. The different kinds of declarative data integrity constraints correspond to the three
fundamental aspects of data integrity: entity integrity, referential integrity, and domain integrity.

A central tenet of relational database theory is that every tuple of every relation (more colloquially, every row of
every table) can be uniquely identified—a condition known as entity integrity. The attribute or combination of
attributes (the column or combination of columns) that ensures uniqueness is known as the primary key. A table can
have only one primary key. SQL Server allows you to designate the columns that make up the primary key when you
define a table. This is known as a PRIMARY KEY constraint. SQL Server uses this PRIMARY KEY constraint to
guarantee that the uniqueness of the values in the designated columns is never violated. It enforces entity integrity for a
table by making sure that it has a primary key.

Sometimes more than one column (or combination of columns) of a table can uniquely identify a row—for example,
an employee table might have an employee ID (emp_id) column and a Social Security number (soc_sec_num)
column, both of whose values are considered unique. Such columns are often referred to as alternate keys or
candidate keys. These keys must also be unique. Although a table can have only one primary key, it can have
multiple alternate keys. SQL Server supports the multiple alternate key concept via UNIQUE constraints. When a
column or combination of columns is declared unique, SQL Server prevents any row from being added or updated
that would violate this uniqueness.

Assigning an arbitrary unique number as the primary key when no natural or convenient key exists is often most
efficient. For example, businesses commonly use customer numbers or account numbers as unique identifiers or
primary keys. SQL Server makes it easy to efficiently generate unique numbers by allowing one column in a table to
have the IDENTITY property. You use the IDENTITY property to make sure that each value in the column is unique
and that the values will increment (or decrement) by the amount you specify from a starting point that you specify.
(A column with the IDENTITY property typically also has a PRIMARY KEY or UNIQUE constraint, but this isn't
required.)
This document is created with the unregistered version of CHM2PDF Pilot

The second type of data integrity is referential integrity. SQL Server enforces logical relationships between tables
with FOREIGN KEY constraints. A foreign key in a table is a column or combination of columns that match the
primary key (or possibly an alternate key) of another table. The logical relationship between those two tables is the
basis of the relational model; referential integrity means that this relationship is never violated.

For instance, the simple SELECT example shown earlier in the chapter includes a titles table and a publishers table.
In the titles table, the column title_id (title ID) is the primary key. In the publishers table, the column pub_id
(publisher ID) is the primary key. The titles table also includes a pub_id column, which isn't the primary key because
a publisher can publish multiple titles. Instead, pub_id is a foreign key, and it references the primary key of the
publishers table. If you declare this relationship when you define the table, SQL Server enforces it from both sides.
First, it ensures that a title can't be entered in the titles table, or an existing pub_id in the titles table can't be modified,
unless a valid publisher ID for the new pub_id is in the publishers table. Second, it ensures that no changes are made
to a pub_id value in the publishers table without consideration of the referencing values in the titles table. It can do
this in two ways. It can restrict changes to the publishers table so that a publisher can't be deleted or a pub_id value
can't be changed if any titles in the titles table reference that publisher. Alternatively, it can cascade the changes
through the titles table. So, if you were to delete a publisher, all rows in titles that reference that publisher would also
be deleted. If you were to update a pub_id, all matching pub_id values in titles would be updated to the same value.

The third type of data integrity is domain integrity, in which individual data values must meet certain criteria. SQL
Server makes sure that any data entered matches the type and range of the specified datatype. For example, it
prohibits NULL values from being entered in a column unless the column explicitly allows NULLs. SQL Server
supports a wide range of datatypes, resulting in great flexibility and efficient storage. It also supports the definition of
CHECK constraints to define conditions that the data in a column must meet.

Datatypes
SQL Server datatypes provide the simplest form of data integrity by restricting the types of information (for example,
characters, numbers, or dates) that can be stored in the columns of the database tables. You can also design your
own datatypes (user-defined datatypes) to supplement those supplied by the system. For example, you can define a
state_code datatype as two characters (CHAR(2)); SQL Server will then accept only two-character state codes.
You can use a user-defined datatype to define columns in any table. An advantage of user-defined datatypes is that
you can bind rules and defaults (which I'll discuss in the next two sections) to them for use in multiple tables,
eliminating the need to include these types of checks in the front-end application.

CHECK Constraints and Rules

CHECK constraints and rules are integrity constraints that go beyond those implied by a column's datatype. When a
user enters a value, SQL Server checks that value against any CHECK constraint or rule created for the specified
column to ensure that only values that adhere to the definition of the constraint or rule are accepted.

Although CHECK constraints and rules are essentially equivalent in functionality, CHECK constraints are easier to
use and provide more flexibility. CHECK constraints are the preferred mechanism for restricting values in a column;
SQL Server provides rules primarily as backward compatibility feature. A CHECK constraint can be conveniently
defined when a column is defined, it can be defined on multiple columns, and it has access to all of SQL Server's
built-in functions. A rule, however, must be defined and then bound separately to a single column or user-defined
datatype, and it has access only to built-in functions that don't reference database objects.

Both CHECK constraints and rules can require that a value fall within a particular range, match a particular pattern,
or match one of the entries in a specified list. An advantage of CHECK constraints is that they can depend on either
the value of another column or columns in the row or on the value returned by one of the built-in functions. A rule
can't reference other fields. As an example of applying a CHECK constraint or rule, a database containing information
on senior citizens could have the CHECK constraint or the rule "age column must contain a value between 65 and
120 years." A birth certificate database could require that the date in the birth_date column be some date prior to the
This document is created with the unregistered version of CHM2PDF Pilot

current date.

Defaults
Defaults allow you to specify a value that SQL Server inserts if no value is explicitly entered in a particular field. For
example, you can set the current date as the default value for an order_date field in a customer order record. Then, if
a user or front-end application doesn't make an entry in the order_date field, SQL Server automatically inserts the
current date. You can also use the keyword DEFAULT as a placeholder in an INSERT or UPDATE statement,
which instructs SQL Server to set the value to the declared default value.

Triggers
Triggers are a special type of stored procedure. Stored procedures can be executed only when explicitly called;
triggers are automatically invoked, or triggered, by SQL Server, and this is their main advantage. Triggers are
associated with particular pieces of data and are called automatically whenever an attempt to modify that data is
made, no matter what causes the modification (user entry or an application action).

Conceptually, a trigger is similar to a CHECK constraint or rule. SQL Server automatically activates triggers,
constraints, and rules when an attempt is made to modify the data they protect. CHECK constraints and rules then
perform fairly simple types of checks on the data—for example, "make sure the age field has a value between 0 and
120." Triggers, on the other hand, can enforce extremely elaborate restrictions on the data, which helps to ensure that
the rules by which your business operates can't be subverted. Because triggers are a form of stored procedure, they
have the full power of the Transact-SQL language at their disposal and they can invoke other stored and extended
stored procedures. You can write a trigger that enforces complex business rules, such as the following:

Don't accept an order:

If the customer has any past due accounts with us

OR

If the customer has a bad credit rating by ACME Credit Service (with the trigger calling an extended
procedure that automatically dials up ACME to get the credit rating)

OR

If the order is for more than $50,000 and the customer has had an account with us for less than six months

This integrity check is quite powerful. Yet the trigger to enforce it is simple to write. Triggers can also enforce
referential integrity, ensuring that relationships between tables are maintained. For example, a trigger can prohibit a
customer record from being deleted if open orders exist for the customer, or it can prohibit any new order for a
customer for which no record exists. While you can do all this by declaring primary and foreign key constraints,
triggers give you some added flexibility and functionality. Triggers allow you to supply your own error messages,
which might be more useful than the default messages provided by SQL Server. Triggers also allow you to implement
other forms of referential actions beside those available with the foreign key definition. I'll talk more about referential
actions in Chapter 6, where I'll talk about constraints, and in Chapter 12, where I'll go into greater detail about
triggers.

Triggers automatically execute whenever a specified change to a data object is attempted. A trigger executes once
per statement, even if multiple rows would be affected. It has access to the before and after images of the data. SQL
Server 2000 provides two kinds of triggers: "after" triggers and "instead-of" triggers. Instead-of triggers are what
some people might call "before" triggers, but that name is not really accurate for SQL Server. An instead-of trigger
defines actions to be carried out instead of the requested data modification. A before trigger would define actions to
This document is created with the unregistered version of CHM2PDF Pilot

perform prior to the data modification. By default, if a table or view has an instead-of trigger defined for a particular
modification operation (INSERT, UPDATE, or DELETE), that data modification does not take place. Instead, the
trigger action is executed.

An after trigger executes after the data has been modified. The trigger takes further action such as rolling back the
current transaction or carrying out special referential actions.
This document is created with the unregistered version of CHM2PDF Pilot

34

Transaction Processing
Transaction processing guarantees the consistency and recoverability of SQL Server databases. A transaction is the
basic unit of work in SQL Server. Typically, it consists of several SQL commands that read and update the database,
but the update is not considered "final" until a COMMIT command is issued.

Transaction processing in SQL Server ensures that all transactions are performed as a single unit of work—even in
the presence of a hardware or general system failure. Such transactions are referred to as having the ACID
properties: atomicity, consistency, isolation, and durability. In addition to guaranteeing that explicit multistatement
transactions (such as those provided in the following DEBIT_CREDIT example) maintain the ACID properties, SQL
Server guarantees that a single command that affects multiple rows also maintains the ACID properties.

Here's an example in pseudocode of an ACID transaction. (Note that error handling would be required to achieve
the behavior described earlier.)
BEGIN TRANSACTION DEBIT_CREDIT
Debit savings account $1000
Credit checking account $1000
COMMIT TRANSACTION DEBIT_CREDIT

Now let's take a closer look at each of the ACID properties.

Atomicity
SQL Server guarantees the atomicity of its transactions. Atomicity means that each transaction is treated as
all-or-nothing—it either commits or aborts. If a transaction commits, all of its effects remain. If it aborts, all of its
effects are undone. In the previous DEBIT_CREDIT example, if the savings account debit is reflected in the database
but the checking account credit isn't, funds will essentially disappear from the database; that is, funds will be
subtracted from the savings account but never added to the checking account. If the reverse occurs (if the checking
account is credited and the savings account is not debited), the customer's checking account will mysteriously
increase in value without a corresponding customer cash deposit or account transfer. Because of SQL Server's
atomicity feature, both the debit and credit must be completed or else neither event is completed.

Consistency
The consistency property ensures that a transaction won't allow the system to arrive at an incorrect logical state—the
data must always be logically correct. Constraints and rules are honored even in the event of a system failure. In the
DEBIT_CREDIT example, the logical rule is that money can't be created or destroyed—a corresponding,
counter-balancing entry must be made for each entry. (Consistency is implied by, and in most situations is redundant
with, atomicity, isolation, and durability.)

Isolation
Isolation separates concurrent transactions from the updates of other incomplete transactions. In the
DEBIT_CREDIT example, another transaction can't see the work-in-progress while the transaction is being carried
out. For example, if another transaction reads the balance of the savings account after the debit occurs, and then the
DEBIT_CREDIT transaction is aborted, the other transaction will be working from a balance that never logically
This document is created with the unregistered version of CHM2PDF Pilot

existed.

Isolation among transactions is accomplished automatically by SQL Server. It locks data to allow multiple concurrent
users to work with data, but it prevents side effects that can distort the results and make them different than would be
expected if users serialized their requests (that is, if requests were queued and serviced one at a time). This
serializability feature is one of the isolation levels that SQL Server supports. SQL Server supports multiple degrees of
isolation levels that allow you to make the appropriate tradeoff between how much data to lock and how long to hold
locks. This tradeoff is known as concurrency vs. consistency. Locking reduces concurrency (because locked data is
unavailable to other users), but it provides the benefit of higher consistency. (Chapter 14 provides much more detail
about locking.)

Durability
After a transaction commits, SQL Server's durability property ensures that the effects of the transaction persist even
if a system failure occurs. Conversely, if a system failure occurs while a transaction is in progress, the transaction is
completely undone, leaving no partial effects on the data. For example, if a power outage occurs in the midst of a
transaction before the transaction is committed, the entire transaction is automatically rolled back when the system is
restarted. If the power fails immediately after the acknowledgment of the commit is sent to the calling application, the
transaction is guaranteed to exist in the database. Write-ahead logging and automatic rollback and rollforward of
transactions during the recovery phase of SQL Server startup ensure durability.
This document is created with the unregistered version of CHM2PDF Pilot

34

Symmetric Server Architecture

SQL Server uses a single-process, multithreaded architecture known as Symmetric Server Architecture that provides
scalable high performance with efficient use of system resources. With Symmetric Server Architecture, only one
memory address space is provided for the DBMS, eliminating the overhead of having to manage shared memory.

Traditional Process/Thread Model

To understand the architecture of SQL Server, it's useful to first understand the traditional architectures used by
UNIX-based DBMS products. UNIX-based DBMS products are usually structured in one of two ways. One
architecture uses multiple processes (or shadow processes), with one process per user, which makes the system
quite resource-intensive. The second type of architecture employs a single process that tries to simulate an operating
system threading facility by moving in a round-robin way among multiple requests, maintaining a stack for each
request and switching to the specific stack for whatever unit is being executed.
NOTE
A stack is a last-in, first-out (LIFO) data structure that's kept in memory. It essentially serves as the control block for
the executable unit on an operating system. (In Windows NT/2000, the basic executable unit is known as a thread,
often called a lightweight process in other operating systems.) A thread stack stores status data such as function call
return addresses, passed parameters, and some local variables.

In the first approach, because each process has its own address space, processes must resort to shared memory to
communicate with one another. Unfortunately, shared memory is less efficient to use than the private memory of a
process's own address space because of the weight of synchronization mechanisms (semaphores, mutexes, and so
on) that are needed to avoid collisions while accessing shared memory. In addition, the implementation of stack
switching and synchronized access to shared memory adds overhead and complexity. Adding complexity to a system
is never good. The best way to avoid bugs in software and maximize performance is to keep code simple and, better
yet, to write no new code when an existing tried-and-true service exists.

In the second approach, called simulated multithreading, the DBMS performs duties usually handled by the operating
system. Typically, using such an architecture requires that the executing task be trusted to yield back to the system so
another task can be run. If the task doesn't yield (because of software or hardware failure), all other tasks are
severely—and perhaps fatally—affected.

SQL Server Process/Thread Model

SQL Server 2000 uses a scheduling architecture that's somewhere in between the two traditional approaches.
Windows NT/2000 makes available support for lightweight threads, called fibers. Certain high-volume symmetric
multiprocessing (SMP) systems can gain considerable performance benefits by using fibers instead of using threads,
and Microsoft wanted to make this functionality possible with SQL Server.
NOTE
A more complete description of the Windows NT/2000 process and thread model is beyond the scope of this book.
For more information, see Inside Microsoft Windows 2000, Third Edition, by David Solomon and Mark
Russinovich (Microsoft Press, 2000) or Programming Server-Side Applications for Microsoft Windows 2000 by
Jeffrey Richter (Microsoft Press, 2000).

To make the best use of fibers, SQL Server takes control of their scheduling; it does so by using one or more
operating system threads as its own schedulers. These threads are called UMS threads, or User Mode Schedulers.
This document is created with the unregistered version of CHM2PDF Pilot

Whether it's running in thread mode or fiber mode, SQL Server uses only one UMS thread per CPU. Each SQL
Server fiber is associated with exactly one UMS thread, and the fiber remains with that UMS thread for its lifetime.
Windows NT/2000 can schedule each of the UMS threads on any of the available processors. Each UMS thread
determines which of the SQL Server processes associated with that UMS thread should run at any given time.
Chapter 17 examines in detail how to control whether SQL Server processes run on threads or fibers, and the
benefits and drawbacks of each option.

SQL Server processes that execute non_SQL Server code aren't scheduled by the UMS. These processes include
those that run extended stored procedures and those that run distributed queries. These processes are scheduled on
normal operating system threads and managed by Windows NT/2000's own scheduler. In addition, when SQL
Server is executing a backup or restore operation, additional UMS threads control the progress of the operation.

Multiuser Performance
The efficiency of the SQL Server threading model is borne out by its multiuser performance. SQL Server can
efficiently handle hundreds, even thousands, of simultaneous active users. Built-in thread pooling allows workloads of
this magnitude without the need for an external Transaction Processing (TP) Monitor, which adds cost and complexity
to a system.
NOTE
Of course, questions such as "How many users can SQL Server handle?" and "How big a database can it handle?"
never have simple answers. The answers depend on the application and its design, required response times and
throughput, and the hardware on which the system is running.

Most systems that just a couple of years ago required a mainframe or large minicomputer-based solution can now be
efficiently built, deployed, and managed using SQL Server. Such industry-standard benchmarks as TPC-C can be
illuminating. Today's SQL Server can perform workloads that surpass those submitted by the largest mainframe
systems of a few years ago. As computer resources continue to grow, SQL Server will extend its reach into systems
that have traditionally required a mainframe solution.
This document is created with the unregistered version of CHM2PDF Pilot

34

Security
SQL Server 2000 security provides two basic methods for authenticating logins: Windows Authentication and SQL
Server Authentication. In Windows Authentication, SQL Server login security is integrated directly with Windows
NT/2000 security, allowing the operating system to authenticate SQL Server users. Windows Authentication allows
SQL Server to take advantage of the security features of the operating system, such as password encryption,
password aging, and minimum and maximum length restrictions on passwords. In SQL Server Authentication, an
administrator creates SQL Server login accounts within SQL Server, and any user connecting to SQL Server must
supply a valid SQL Server login name and password.

Windows Authentication makes use of trusted connections, which rely on the impersonation feature of Windows
NT/2000. Through impersonation, SQL Server can take on the security context of the Windows NT/2000 user
account initiating the connection and test whether the security identifier (SID) has a valid privilege level. Windows
NT/2000 impersonation and trusted connections are supported by any of the available network libraries
(Net-Libraries) when connecting to SQL Server running under Windows NT/2000.

Under Windows 2000 only, SQL Server 2000 can use Kerberos to support mutual authentication between the client
and the server, as well as the ability to pass a client's security credentials between computers so that work on a
remote server can proceed using the credentials of the impersonated client. With Windows 2000, SQL Server 2000
uses Kerberos and delegation to support Windows Authentication as well as SQL Server Authentication.

The authentication method (or methods) used by SQL Server is determined by its security mode. SQL Server can
run in one of two different security modes: Windows Authentication Mode (which uses Windows Authentication) and
Mixed Mode (which uses both Windows Authentication and SQL Server Authentication). Windows Authentication
Mode is available only for instances of SQL Server running on Windows NT/2000. When you connect to an instance
of SQL Server configured for Windows Authentication Mode, you cannot supply a SQL Server login name and your
Windows NT/2000 username determines your level of access to SQL Server.

Under Mixed Mode, Windows NT/2000-based clients can connect using Windows Authentication, and connections
that don't come from Windows NT/2000 clients or that come across the Internet can connect using SQL Server
Authentication. In addition, when a user connects to an instance of SQL Server that has been installed in Mixed
Mode, the connection can always explicitly supply a SQL Server login name. This allows a connection to be made
using a login name distinct from the username in Windows NT/2000. Because Windows 98 doesn't support Windows
Authentication, SQL Server on Windows 98 runs only in Mixed Mode and only supports SQL Server
Authentication.

Monitoring and Managing Security

SQL Server makes it easy to monitor logon successes and failures. Using SQL Enterprise Manager, administrators
can simply select the appropriate Audit Level option on the Security tab of the Properties dialog box for a particular
installation of SQL Server. When logon monitoring is enabled in this way, each time a user successfully or
unsuccessfully attempts to log on to SQL Server, a message is written to the Windows NT/2000 event log, the SQL
Server error log, or both, indicating the time, date, and user who tried to log on.

SQL Server has a number of facilities for managing data security. You can grant or deny access privileges (select,
insert, update, and delete) to users or groups of users for objects such as tables and views. You can grant execute
privileges for normal and extended stored procedures. For example, to prevent a user from directly updating a
specific table, you can write a stored procedure that updates the table and then cascades those updates to other
tables as necessary. You can grant the user access to execute the stored procedure, thereby ensuring that all updates
This document is created with the unregistered version of CHM2PDF Pilot

take place through the stored procedure, eliminating the possibility of integrity problems arising from ad hoc updates
to the base table.
This document is created with the unregistered version of CHM2PDF Pilot

34

High Availability
In many mission-critical environments, the application must be available at all times—24 hours a day, seven days a
week. SQL Server helps ensure availability by providing online backup, online maintenance, automatic recovery, and
the ability to install SQL Server on a cluster for failover support. SQL Server's dynamic online backup allows
databases to be backed up while users are actively querying and updating the database. The SQL Server Agent
service provides a built-in scheduling engine that enables backups to occur automatically, without administrator
involvement. You can also accomplish other maintenance tasks, such as diagnostics, design changes (for example,
adding a column to a table), and integrity changes without having to shut down SQL Server or restrict user access.

Only a few systemwide configuration modifications, such as changing the maximum number of user connections,
require that SQL Server be restarted. Although reconfiguration doesn't commonly occur in a well-planned and
well-deployed production system, it can typically be completed with less than a minute of system downtime if it's
necessary.

In the event of a system failure, such as a power outage, SQL Server ensures rapid database recovery when services
are restored. By using the transaction logs associated with each database, SQL Server quickly recovers each
database upon startup, rolling back transactions that weren't completed and rolling forward transactions that were
committed but not yet written to disk. In addition, you can set the SQL Server Agent service to continually monitor
the state of SQL Server. If an error occurs that causes SQL Server to stop unexpectedly, the service detects this and
can automatically restart SQL Server with minimal interruption.

In cooperation with shared-disk cluster hardware, SQL Server Enterprise Edition provides failover support when
you install one or more instances of SQL Server 2000 on a node that's part of a Microsoft Cluster Service (MSCS)
cluster. Up to four nodes can be part of a cluster. The applications communicating with SQL Server don't need to
know that the server they're connecting to is actually a virtual server, which can be running on any member of a
cluster. If the server on which SQL Server is running fails, one of the other servers in the cluster immediately takes
ownership of the disks that hold the SQL Server data and takes over the failed server's workload. The other servers
in the cluster don't need to be inactive when the first server is working—they can run SQL Server while it performs
other activities.
This document is created with the unregistered version of CHM2PDF Pilot

34

Distributed Data Processing

SQL Server provides features such as linked servers, remote procedure calls, and a two-phase commit protocol that
enable you to easily manage and use data in distributed environments. Microsoft Distributed Transaction Coordinator
(MS DTC) is the vote collector and coordinator of transactions, and it allows many types of systems to participate,
laying the foundation for ACID transactions among heterogeneous systems.

A system participating in a transaction coordinated by MS DTC manages its own work and is called a resource
manager. This resource manager system communicates with MS DTC, which coordinates all the resource managers
participating in the transaction to implement the two-phase commit protocol. Distributed transactions honoring the
ACID properties are supported as a whole: the entire distributed transaction at all sites either commits or aborts.

In the first phase of the two-phase commit protocol, all participating resource managers (that is, those that have
enlisted in the transaction) prepare to commit. This means that they have acquired all the locks and resources they
need to complete the transaction. MS DTC then acts as a vote collector. If it gets confirmation that all participants are
prepared to commit, it signals to go ahead and commit.

The actual COMMIT is the second phase of the protocol. If one or more participants notify the system that they
can't successfully prepare the transaction, MS DTC automatically sends a message to all participants indicating that
they must abort the transaction. (In this case, an abort, rather than a commit, is the protocol's second phase.) If one
or more participants don't report back to MS DTC in phase one, the resource managers that have indicated they're
prepared to commit (but haven't committed because they haven't yet received the instruction to do so) are said to be
in doubt. Resource managers that have transactions in doubt hold the necessary locks and resources until the
transaction either commits or aborts, thereby maintaining the ACID properties. (SQL Server provides a way to force
in-doubt transactions to abort.)

Another important distributed capability allows SQL Server to issue a remote procedure call (RPC) to another
server running SQL Server. RPCs are stored procedures that can be invoked from a remote server, allowing
server-to-server communication. This communication can be transparent to the client application because the client
can execute a procedure on one server and that procedure can then invoke a procedure on a different server.

By using RPCs, you can easily extend the capacity of an application without the added cost of reengineering the client
application. RPCs can also be coordinated by the MS DTC service to ensure that the transactions maintain their
ACID properties. The default behavior is to not execute the RPC in the context of a transaction so that if the local
transaction is rolled back, work done by the RPC will still be committed. You can override this behavior by using the
command BEGIN DISTRIBUTED TRANSACTION or by setting the configuration option remote proc trans to 1.

You can also manage distributed data by defining and accessing linked servers. A linked server can be any data
source for which an OLE DB provider exists. The most commonly used OLE DB providers are the SQL Server
OLE DB provider; the Jet provider, which allows you to connect to Microsoft Access databases, Excel
spreadsheets, and text files; and the Open Database Connectivity (ODBC) provider, which allows you to connect to
any ODBC data source.

A linked server is a logical name defined using the sp_addlinkedserver stored procedure. The information you
provide when you define the linked server includes the name and location of the data source to connect to. Once
you've defined the name, you can use that name as the first part of a four-part name for any object, such as a table, a
view, or a stored procedure. Objects on the linked server can be queried as well as modified, and they can be used in
joins with local objects. The types of queries possible on the linked server depend on the capabilities of the particular
OLE DB providers. Some providers, such as the one for SQL Server, allow all data modification operations and full
transaction control across linked servers. Others, such as the OLE DB provider for text files, allow only querying of
This document is created with the unregistered version of CHM2PDF Pilot

data.
This document is created with the unregistered version of CHM2PDF Pilot

34

Data Replication
Replication allows you to automatically distribute copies of data from one server to one or more destination servers
at one or more remote locations. Data integrity is a key design point of SQL Server's replication capabilities. The
data at replicating sites might be slightly outdated, but it will accurately reflect the state of the data at a recent point in
time and is guaranteed to reflect any changes made soon after they occur (or soon after a reliable network connection
is established).

SQL Server 2000 supports three modes of replication, which are detailed on the companion CD in a whitepaper
titled "Understanding SQL Server 7.0 Replication."
 Transactional replication In this mode of replication, one site is considered the owner, or publisher, of a
published article (a table or a subset of a table), and other sites subscribe to that article. All changes made to
the article must be made at the publisher and then replicated to the subscribers.
 Merge replication Although the publisher defines the articles to be published, any participating sites can
make changes. System tables keep track of the changes, which are then propagated to all participants,
including the original publisher. You can define which type of conflict resolution to use in the event of
conflicting updates made by multiple sites.
 Snapshot replication All data from the publishing site is periodically copied to all the subscribing sites.
Individual changes aren't propagated. This method was formerly called scheduled table refresh.

Distributed transactions that use the two-phase commit protocol guarantee that the ACID properties are maintained,
but replication doesn't. Replication isn't strongly consistent (the C in ACID). Instead, it provides loosely consistent
data. With the two-phase commit protocol, a transaction is an all-or-nothing proposition, and the data is guaranteed
to be strongly consistent. But inherent in the two-phase commit algorithm is the fact that a failure at any one site
makes the entire transaction fail or can keep the transaction in doubt for a long period of time, during which all
participants must hold locks, thereby crippling concurrency. SQL Server does provide an option to transactional
replication called Immediate Updating Subscribers that uses distributed transactions between one subscriber and the
publisher. Other subscribers to the same publication will not be updated as part of the transaction, however.

At first glance, you might think a system should require that updates be made at all sites in real time. In fact, when
the costs of two-phase commit are realized (chiefly, the vulnerability that can result from a failure at just one node), the
most pragmatic solution might be to make changes in real-enough time.

For example, suppose you run a car rental agency with 500 rental counters worldwide and you maintain a customer
profile table containing 500,000 renters who belong to your Gold Card program. You want to store this customer
profile table locally at all 500 rental counters so that even if a communication failure occurs, the profiles will be
available wherever a Gold Card member might walk up to do business. Although all sites should have up-to-date
records of all customers, it would be disastrous to insist that an update of the Gold Card profile must occur as part of
a two-phase transaction for all 500 sites or not at all. If you were to do that, because of the vagaries of worldwide
communication or because a storm might knock out the power at one site, you wouldn't be able to perform a simple
update to the customer profile very often.

Replication is a much better solution in this case. The master customer profile table is maintained at your corporate
headquarters. Replication publishes this data, and the rental counters subscribe to this information. When customer
data is changed or when a customer is added or removed, these changes (and only these changes) are propagated to
all the subscribing sites. In a well-connected network, the time delay might be just a few seconds. If a particular site is
unavailable, no other sites are affected—they still get their changes. When the unavailable site is back online, the
changes are automatically propagated and the subscriber is brought up to date. At any time, a given rental counter
might have information that's slightly out of date—that is, it might reflect the state of the data at the corporate
headquarters at some earlier time. This is considered loosely consistent, as opposed to the strongly consistent model
This document is created with the unregistered version of CHM2PDF Pilot

of two-phase commit, in which all sites (or none) immediately reflect the change.

Although a time delay can occur in loosely consistent systems, maintaining transactional consistency is one of the chief
design points of SQL Server replication. If multiple updates occur as a single atomic transaction to data being
replicated, the entire transaction is also replicated. At the subscribing site, the transaction either entirely commits or is
again replicated until it commits.

With SQL Server, data can be replicated continuously or at specified intervals. It can be replicated in its entirety or
as filtered subsets (known as horizontal and vertical partitions). In addition to replicating to other SQL Servers,
SQL Server 2000 can replicate to heterogeneous data sources that have an appropriate ODBC driver or OLE DB
provider available. These include Access databases, Oracle databases, and other ODBC data sources. SQL Server
2000 can also replicate from other data sources, using ODBC drivers or OLE DB providers. These sources include
Oracle 8, Microsoft Jet version 4, and IBM DB/2.

Unlike SQL Server, some products on the market promote replication as an "update anywhere, anytime, any way"
model. However, this model has inherently unstable behavior if many nodes participate and update activity is
moderate to heavy.1

Updates made at multiple sites will conflict with one another and will have to be reconciled. SQL Server 2000's
merge replication does allow multiple site updates, but it's not a true "update anywhere, anytime, any way" solution.
(See the whitepaper on the companion CD for more details on exactly what SQL Server merge replication can do.)
This document is created with the unregistered version of CHM2PDF Pilot

34

Systems Management
The difficulty of systems management is probably the single biggest obstacle to mass deployment of client/server
solutions. Far from being a downsized version of the mainframe, today's distributed client/server system can be
deployed on dozens or even hundreds of distributed servers, all of which must be controlled to the same exacting
standards as mainframe production software systems. The issues here reside both inside and outside the database
environment. SQL Server provides a comprehensive architecture and tools for managing the database and related
activities.

SQL Server Enterprise Manager

SQL Server Enterprise Manager is a major advancement in making client/server deployments manageable. In SQL
Server 2000, SQL Server Enterprise Manager is a snap-in to Microsoft Management Console (MMC), a tool that
provides a common interface for managing various server applications in a Windows network. (Another snap-in that's
provided on your SQL Server 2000 CD for use with MMC is the SQL Server OLAP Services snap-in.)

The easy-to-use SQL Server Enterprise Manager supports centralized management of all aspects of multiple
installations of SQL Server, including security, events, alerts, scheduling, backup, server configuration, tuning, and
replication. SQL Server Enterprise Manager allows you to create, modify, and copy SQL Server database schemas
and objects such as tables, views, and triggers. Because multiple installations of SQL Server can be organized into
groups and treated as a unit, SQL Server Enterprise Manager can manage hundreds of servers simultaneously.

Although it can run on the same computer as the SQL Server engine, SQL Server Enterprise Manager offers the
same management capabilities while running on any Windows NT/2000-based machine. SQL Server Enterprise
Manager also runs on Windows 98, although a few capabilities aren't available in this environment (most notably the
ability to use the Service Control Manager, a feature of Windows NT/2000, to remotely start and stop SQL Server).
In addition, the efficient client/server architecture of SQL Server makes it practical to use the remote access (dial-up
networking) capabilities of Windows NT/2000 as well as Windows 98 for administration and management.

Enterprise Manager's easy-to-use interface is shown in Figure 2-2. You can perform even complex tasks with just a
few mouse clicks.

Figure 2-2. The SQL Server Enterprise Manager interface.

This document is created with the unregistered version of CHM2PDF Pilot

SQL Server Enterprise Manager relieves you from having to know the specific steps and syntax to complete a job. It
provides more than 20 wizards to guide you through the process of setting up and maintaining your installation of SQL
Server. A partial list is shown in Figure 2-3.

Figure 2-3. Some of the wizards available for setting up and maintaining your server.

Distributed Management Objects

Microsoft SQL Server Distributed Management Objects (SQL-DMO) is the SQL Server Automation object model.
You can use the SQL-DMO objects, properties, methods, and collections to write scripts and programs that
administer multiple servers running SQL Server distributed across a network. SQL Server Enterprise Manager is built
entirely with SQL-DMO. You can customize your own specific management needs using SQL-DMO, or you can
integrate management of SQL Server into other tools you use or provide.

All SQL Server functions are exposed through the SQL-DMO object model. The primary object is SQLServer,
which contains a collection of Database objects. Each Database object contains a collection of Table, View, and
StoredProcedure objects. You can control SQL Server by changing SQL-DMO object properties (
SQLServer.Name = "MARKETING_SVR") and by invoking SQL-DMO object methods (SQLServer.Start or
SQLServer.Shutdown).

Table 2-1 provides some examples of SQL-DMO objects and methods.

Table 2-1.SQL-DMO objects and methods.

Object.Method Action
SQLServer.Shutdown Stops SQL Server.
SQLServer.Start Starts SQL Server.
Index.UpdateStatistics Updates optimizer information for indexes.
This document is created with the unregistered version of CHM2PDF Pilot

Database.Tables.Add Adds a table to a database.

The SQL-DMO object model is comprehensive, consisting of dozens of distinct objects and hundreds of Component
Object Model (COM) interfaces. The organization of these objects greatly simplifies the task of learning and fully
using SQL Server management components.

Any Automation-controlling application can harness the power and services of SQL-DMO. Probably the most
common such Automation controller is Microsoft Visual Basic.

Windows Management Instrumentation

The SQL Server 2000 CD contains a technology preview of a new API for supporting your SQL Server installations
called Windows Management Instrumentation (WMI). WMI, which is similar to SQL-DMO and in fact maps directly
to SQL-DMO methods and objects, lets you access and manipulate not only your SQL Server objects but also other
components in your Windows network. These other components can include servers, printers, registry keys,
performance monitor counters (system monitor counters in Windows 2000), and events from Windows NT/2000's
application log, system log, and security log.

WMI is not installed by the SQL Server 2000 installation program but is available in the \x86\other\wmi folder on the
same CD. Unlike SQL-DMO, the WMI components are installed on the server instead of on the client machines. In
future releases, WMI will communicate directly with SQL Server rather than going through an SQL-DMO layer.

SQL-DMO and Visual Basic Scripting

The power of using the SQL-DMO object model for SQL Server management becomes clear when you consider
the potential of using a rapid application development (RAD) language such as Visual Basic as a scripting environment
for administrative tasks. The following sample code lists the name of and space available on all databases on a server.
This code is simple, compact, and easy to write and read, yet it's powerful. (Traditionally, programming such a task
would have required several pages of much more complex C code.)
Dim MyServer As New SQLServer ' Declare the SQLServer object
Dim MyDB As Database

MyServer.Name = "MARKETING_SVR"
MyServer.Login = "sa"
MyServer.Password = "password"
MyServer.Connect ' Connect to the SQL Server

' List the name and space available for all databases
For Each MyDB In MyServer.Databases
Debug.Print MyDB.Name, MyDB.SpaceAvailable
Next MyDB

MyServer.DisConnect ' Disconnect

SQL Server Agent

SQL Server Agent is an active, intelligent agent that plays an integral role in the management of the SQL Server
environment. It provides a full-function scheduling engine that supports regular tasks for the management and
administration of SQL Server, and it allows you to schedule your own jobs and programs.

SQL Server Agent is a Windows NT/2000-based service that can be started when the operating system starts. You
can also control and configure it from within SQL Server Enterprise Manager or SQL Server Service Manager. SQL
This document is created with the unregistered version of CHM2PDF Pilot

Server Agent is entirely driven by entries in SQL Server tables that act as its control block. Clients never directly
communicate with or connect to SQL Server Agent to add or modify scheduled jobs. Instead, they simply make the
appropriate entries in the SQL Server tables (although this typically occurs through SQL Server Enterprise Manager
via a simple dialog box that's similar to a typical calendar program). At startup, SQL Server Agent connects to an
installation of SQL Server that contains its job table and then loads the list of jobs, the steps that make up each job,
and the possible alerts that can be fired.

SQL Server Agent, like the SQL Server engine, is a single multithreaded process. It runs in its own process space
and manages the creation of operating system threads to execute scheduled jobs. Its discrete managed subsystems
(for replication, task management, and event alerting) are responsible for all aspects of processing specific jobs. When
jobs are completed (successfully or not), the subsystem returns a result status (with optional messages) to SQL
Server Agent. SQL Server Agent then records the completion status in both the operating system's application log
and the job history table in SQL Server and optionally sends an e-mail report of the job status to the designated
operator.

In SQL Server 2000, scheduled jobs can be made up of multiple steps; each step defines the action to take if it
succeeds and the action to take if it fails. Thus, you can control the execution of subsequent steps when a step fails.

SQL Server 2000 allows you to define a server as a master server for the purpose of scheduling jobs that will be
carried out by one or more target servers. These target servers periodically poll their master server to determine
whether any new jobs have been defined. If so, the master server downloads the job definitions to the target server.
After the jobs are run on the target server, the success or failure status is reported back to the master server. The
master server keeps the complete job history for all its target servers.

The event/alert subsystem gives SQL Server its ability to support proactive management. The primary role of the
event/alert subsystem is to respond to events by raising alerts and invoking responses. As triggering activities (or
user-defined activities) occur in the system, an event is posted to the operating system application log. SQL Server
Agent monitors the application log to detect when an event has occurred.

SQL Server Agent determines whether any alerts have been defined for this event by examining the event's error
number, severity, database of origin, and message text. If an alert has been defined (in the alert table), the operator(s)
can be alerted via e-mail or pager or with a net send message. Alternatively, a SQL Server Agent job can be invoked
when the alert is fired and can take corrective action. (For example, SQL Server Agent might automatically expand a
full database.)

If no alerts have been defined locally, the event can be forwarded to another server for processing. This feature
allows groups of servers to be monitored centrally so that alerts and operators can be defined once and then applied
to multiple servers. Beyond the database environment, Microsoft Systems Management Server (SMS)—a
BackOffice component—is available to provide key services to manage the overall software configuration of all the
desktops and servers in the environment.

Alerts can also be fired in response to System Monitor (or Performance Monitor) thresholds being crossed.
Normally, the crossing of a threshold isn't considered an error, and no event is written to the application log.
However, the alert system in SQL Server 2000 is integrated with the performance counters available through System
Monitor (discussed later in this chapter), and SQL Server can take action when a counter rises above or falls below a
specified value. The kinds of actions possible are the same actions described previously: the operator(s) can be
alerted via e-mail or pager or with a net send message. Alternatively, a SQL Server Agent job can be invoked to
take corrective action when the alert is fired.

For example, SQL Server Agent might automatically expand a database that's almost full. Note that being almost full
is technically not an error: no error number or severity level is associated with this situation. However, because
Performance Monitor can detect the condition of a database file reaching a certain percentage of its capacity, an alert
can be raised. Figure 2-4 shows the definition of an alert based on a Performance Monitor threshold. This alert is
fired when the log file for the Northwind database exceeds 80 percent of its capacity.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 2-4. Alert definition for a greater-than-80-percent-full condition in the Northwind database log file.
This document is created with the unregistered version of CHM2PDF Pilot

34

SQL Server Utilities and

Extensions
SQL Server also includes utilities and extensions that provide increased functionality, such as Internet enabling,
monitoring capability, easy setup, and easy data importing. (For in-depth information on any of the following utilities
and extensions, see SQL Server Books Online.)

Web Assistant Wizard and Internet Enabling

SQL Server provides dynamic ways to work with the Internet using Web Assistant Wizard and interoperability with
Microsoft Internet Information Services (IIS). Although both Web Assistant Wizard and IIS enable SQL Server data
to be used with Web pages, they satisfy different needs.

Web Assistant Wizard generates HTML files from the result sets of SQL Server queries, making it simple to publish
SQL Server data on the Internet. Let's say, for example, that a parts supplier keeps its inventory list in SQL Server.
The supplier can publish its current parts inventory as a Web page (an HTML file) using Web Assistant Wizard. The
wizard lets you submit an ad hoc query or stored procedure, do some simple formatting, include links to other Web
pages, and use a template for more advanced formatting. The output of the query is written as an HTML table, and a
Web page is created. You can also use Web Assistant Wizard to automatically update the Web page at regular
intervals or whenever the data changes (via a trigger).

Web Assistant Wizard is distinct from but complementary to IIS in BackOffice. With the wizard, users browsing the
Web page work separately from SQL Server because the data on the Web page has been extracted from the
database beforehand. The wizard doesn't use or require IIS, and pages that the wizard generates can be viewed using
any Internet browser.

IIS uses SQL Server's high-performance native ODBC interface to allow SQL queries to be fired from a Web page
when a user accesses a particular region on the page. The results are then dynamically retrieved and combined with
the HTML file for up-to-date viewing. In addition to Web Assistant Wizard and the dynamic query capabilities
enabled with IIS, SQL Server is Internet-enabled in several other important ways. By minimizing network traffic and
handshaking, SQL Server is inherently designed for efficient client/server computing. Extremely rich requests can be
packaged via stored procedures or Transact-SQL batches for resolution entirely at the server, with only the results
sent back to the initiating client application.

This capability has been a hallmark of SQL Server's client/server architecture from the outset, but nowhere is it more
important than on the Internet, where network speed and bandwidth are often quite limited. In addition, SQL Server's
networking architecture allows for ease of use and security on the Internet, including network name resolution. For
example, Internet users can connect to SQL Server via a friendly name such as www.microsoft.com instead of via an
arcane IP address such as 200.154.54.678. Secure encryption of data over the Internet is also possible.

SQL Server 2000 also provides XML support that enables querying and updating of XML documents in a SQL
Server 2000 database. Standard SQL statements with the FOR XML clause can be issued from a Web-based client,
and the results are returned to the client in XML format.

SQL Profiler
This document is created with the unregistered version of CHM2PDF Pilot

SQL Profiler is a graphical utility that allows database administrators and application developers to monitor and
record database activity. SQL Profiler can display all server activity in real time, or it can create filters that focus on
the actions of particular users, applications, or types of commands. SQL Profiler can display any SQL statement or
stored procedure sent to any instance of SQL Server (if your security privileges allow it) as well as the output or
response sent back to the initiating client.

SQL Profiler allows you to drill down even deeper into SQL Server and see every statement executed as part of a
stored procedure, every data modification operation, every lock acquired or released, or every occurrence of a
database file growing automatically. And that's just for starters. Dozens of different events can be captured, and
dozens of data items can be captured for each event.

SQL Profiler is an important tool for tuning and debugging applications and for auditing and profiling the use of SQL
Server. I'll discuss SQL Profiler in more detail in Chapter 17.

SQL Server Service Manager

SQL Server Service Manager (Figure 2-5) manages all instances of SQL Server, SQL Server Agent and MS DTC,
as well as Microsoft Search and MSSQLServerOLAPService services. It provides a simple way to start, stop, or
check the state of any of these services. Once you've used this application after booting the operating system, the
application places an icon on the taskbar in the area of the taskbar clock. You can right-click on this icon to retrieve a
menu of all the tasks that SQL Server Service Manager supports.
NOTE
On machines running Windows 98, SQL Server Service Manager looks as shown in Figure 2-5. However, the
components that can be started aren't services. Machines running Windows 98 don't support services, so SQL
Server Service Manager actually starts separate executable programs. (The Microsoft Search service isn't available at
all on Windows 98.)

Figure 2-5. SQL Server Service Manager.

System Monitor Integration

SQL Server provides an extension DLL (SQLCTR80.DLL) that integrates with the Windows NT/2000
Performance tool and graphically displays many important performance statistics such as memory use, number of
users, transactions per second, and CPU use. Integrating with the Performance tool is advantageous because it allows
you to use a single tool to measure all aspects of a system's performance. If SQL Server simply provided its own
performance-monitoring tool, you'd still have to check the performance of the operating system and network.
Integration with the operating system's Performance tool provides one-stop shopping. A Windows 2000 Performance
tool System Monitor graph is shown in Figure 2-6.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 2-6. Windows 2000 Performance tool.

Using the Performance tool, you can set an alert on any statistic that's being monitored; when a predefined threshold
is reached, the operating system automatically executes a predefined command. As previously mentioned, you can
allow SQL Server Agent to respond directly to Performance tool thresholds being crossed.

Client Network Utility

The Client Network utility is used to configure DB-Library and the Net-Libraries that clients use when connecting to
various SQL Servers. If you don't use DB-Library or you don't do any customization of network connections, you
might not need to use this tool. However, if you want to examine or change the TCP/IP port that your named instance
of SQL Server is listening on, you can do that using the Client Network utility. Also, if you want client tools to
communicate with any local SQL Server instance using the Shared Memory Net-Library, you must make sure that
feature is enabled using the Client Network utility. The Client Network utility also allows you to set up aliases for your
server names. You can set up a simple one-word name to use instead of the two-part name normally required to
access a named instance of SQL Server. I'll show an example of how to do this when I discuss named instances in
detail in Chapter 4.

Server Network Utility

The Server Network utility is used to manage the Net-Libraries on the server machine. Because the Net-Libraries on
which SQL Server listens for client requests are specified during installation, most administrators never need to use
this utility.

SQL Server Installation

SQL Server's graphical installation program allows you to set up SQL Server with a degree of ease and speed
unprecedented for a full-featured DBMS. If you select the defaults, you can install SQL Server in 5 to 10 minutes,
depending on the speed of the computer. A custom installation will typically take well under 30 minutes. You can
easily specify options so that the installation program is fully automated for numerous machines, or you can
encapsulate the installation of SQL Server in the rest of an application's installation process.

OSQL and ISQL

This document is created with the unregistered version of CHM2PDF Pilot

A simple interactive window in which to submit basic SQL commands and get results is a crucial tool for a database
developer—as a hammer is to a carpenter. Even though other, more sophisticated power tools are useful, the basic
tools are essential. SQL Server provides the basic tools in two styles: OSQL and ISQL.

Both OSQL (OSQL.EXE) and ISQL (ISQL.EXE) are character-based command-line utilities. ISQL is based on
the DB-Library interface, and OSQL is ODBC-based. Many parameters, including the SQL statement or the name
of the file containing the statement, can be passed to these character-based utilities. Upon exit, they can return status
values to the Windows client that can be checked within a command file (CMD or BAT). Consequently, programs
commonly launch scripts of SQL commands by spawning one of these character-based utilities and passing the
appropriate parameters and filenames. The SQL Server installation program itself spawns OSQL.EXE numerous
times with scripts that install various database objects and permissions.

SQL Query Analyzer

SQL Query Analyzer (ISQLW.EXE) provides a clean, simple, graphical user interface (GUI) for running SQL
queries and viewing the results. SQL Query Analyzer allows for multiple windows so that simultaneous database
connections (to one or more installations of SQL Server) can exist and be separately sized, tiled, or minimized. You
can choose to have your query results displayed in a text window or in a grid. SQL Query Analyzer provides a
graphical representation of SHOWPLAN, the steps chosen by the optimizer for query execution. It also provides
optional reports of the actual commands processed by SQL Server (a server-side trace) and of the work done by the
client. SQL Server 2000 Query Analyzer comes with an Object Browser that lets you drag and drop object or table
names into your query window and helps you build SELECT, INSERT, UPDATE, or DELETE statements for any
table. SQL Query Analyzer also comes with a T-SQL Debugger tool that I'll describe in more detail in Chapter 16.

SQL Query Analyzer is a wonderfully versatile tool for development and for testing batches and stored procedures
that you'll incorporate into your SQL Server applications. Take some time to just play with the terrific interface so that
you get very comfortable with it. You can use SQL Query Analyzer to:
 Format your SQL queries
 Use templates for your stored procedures, functions, and basic SQL statements
 Drag object names from the Object Browser to the Query window to see their definition
 Define your own hotkeys

Bulk Copy and Data Transformation Services

SQL Server provides a character-based utility called bcp, or bulk copy (BCP.EXE), for flexible importing and
exporting of SQL Server data. Similar in format to ISQL and OSQL, bcp allows parameters to be passed to it and is
often called from command files. A special set of functions exists in ODBC that lets you easily create a custom data
loader for your application; the bcp utility is a generalized wrapper application that calls these functions.

SQL Enterprise Manager provides facilities to simplify the transfer of data into or out of SQL Server, and this
functionality is incorporated into the Data Transformation Services (DTS) tool. Unlike the command-line bcp utility,
DTS allows you to create a new table as you copy data to your SQL Server as well as to easily change column
datatypes or formats. DTS goes far beyond just simple import and export of data to or from SQL Server, however.
You can also use it to move data to or from any OLE DB data source, applying all kinds of data transformations
along the way. Multiple transfers can be defined and stored in a DTS package, which can then be run from a
command line using the dtsrun utility (DTSRUN.EXE).

SNMP Integration
SQL Server provides support for Simple Network Management Protocol (SNMP), a standard protocol within
TCP/IP environments, via SQL Server Management Information Base (MIB), another standard of the SNMP and
TCP/IP environment. A group of database vendors, including Microsoft, cooperated to define a standard MIB that
This document is created with the unregistered version of CHM2PDF Pilot

reports certain status data about a database environment for monitoring purposes.
NOTE
This group of database vendors was a subcommittee of the Internet Engineering Task Force (IETF). The draft
specification is known as IETF SNMP RDBMS-MIB (RFC 1697). SQL Server MIB is generally based on this
proposal but provides additional data beyond that called for in the specification.

For example, status information (such as whether SQL Server is currently running, when it was last started, and how
many users are connected) is reported to SNMP via this MIB. A variety of SNMP management and monitoring tools
exist and can access this data. If, for example, you use Hewlett-Packard's OpenView in managing your network,
SQL Server MIB enables those tools to also monitor multiple statistics regarding SQL Server.

SQL Server Books Online

All SQL Server documentation is available on line as well as in printed form. A powerful viewer for the online
documentation makes it easy to search for topics. Even if you favor printed books, you'll appreciate the speed and
convenience of the SQL Server Books Online search capabilities. Also, because of the lead time required in the
production of the printed manuals that ship with SQL Server, SQL Server Books Online is more complete and
accurate than the manuals. Because this book can't hope to and doesn't try to replace the complete documentation
set, the SQL Server evaluation CD that accompanies this book also contains the complete SQL Server Books
Online documentation. Although you need to install SQL Server to view the complete Books Online, you can view all
the SQL Server Setup documentation right from the initial screen of the installation CD.
This document is created with the unregistered version of CHM2PDF Pilot

34

Client Development Interfaces

SQL Server provides several interfaces that support client application development—ODBC, Remote Data Objects
(RDO), OLE DB, ActiveX Data Objects (ADO), and the legacy interfaces DB-Library and Embedded SQL for C
(ESQL/C). The book's companion CD includes a whitepaper titled "Designing Efficient Applications for SQL Server"
that compares all the interfaces described in this section—and a few more besides.

ODBC
ODBC is an API for database access that's both a formal and a de facto industry standard. Besides being one of the
most popular database interfaces used by applications today, ODBC has gained status as the formal call-level
interface standard by American National Standards Institute (ANSI) and International Organization for
Standardization (ISO). SQL Server provides a high-performance ODBC interface for all Windows-based
programming environments, and it can be distributed royalty-free with any application. The SQL Server ODBC driver
implements every function in the ODBC 3 specification. In ODBC-speak, this makes it fully Level 2 (the highest level)
conformant.

RDO is an object interface that's closely tied to ODBC, which means that it exposes all the functionality in the
ODBC driver and is easily available to Visual Basic programs. RDO supports building visual controls tied directly to
SQL Server data, which greatly reduces the amount of code that must be written to display data on the screen.

OLE DB
OLE DB was first released by Microsoft in 1996 to provide a COM interface to any tabular data source (that is,
data that can be represented with rows and columns). This includes data in spreadsheets and even text files. OLE DB
can be considered an object version of ODBC but is more powerful in that it can access data from data sources
beyond those that ODBC can access. Unlike other object interfaces to SQL Server such as RDO, OLE DB doesn't
make programming a call-level interface like ODBC any easier. Also, because OLE DB uses pointer data types
extensively, it's only accessible from C and C++.

ADO
ADO is a higher-level object interface on top of OLE DB that provides much of the same functionality and
performance. Because ADO is pointerless, it can be accessed from scripting languages such as JScript and
development software such as Visual Basic, as well as from C and C++.

ADO is the recommended and supported interface for Internet applications written using the Microsoft Visual
InterDev development tool. Applications written with Visual InterDev can call ADO from Active Server Pages (ASP)
and incorporate code written in VBScript or JScript.

DB-Library
DB-Library is a SQL Server-specific API that provides all the necessary macros and functions for an application to
open connections, format queries, send them to the server, and process the results. You can write custom DB-Library
applications using C, C++, or Visual Basic (or any programming language capable of calling a C function).
This document is created with the unregistered version of CHM2PDF Pilot

DB-Library is the original SQL Server programming interface. Libraries are provided for MS-DOS, Windows 3.1,
Windows 98, and Windows NT/2000. Developers are granted licensed rights to redistribute the DB-Library runtimes
royalty-free. Although DB-Library continues to be officially supported for SQL Server 2000, it's not guaranteed to
support the full-feature set.

ESQL/C
SQL Server provides an ESQL/C precompiler that allows developers to write SQL Server applications by
embedding the SQL queries directly in their C source code. Many minicomputer and mainframe developers are
already accustomed to this style of programming, and ESQL/C might be a natural choice for that reason. In addition,
Microsoft has licensed some of the ESQL/C run-time environment to MERANT Micro Focus, the leading provider
of COBOL compilers and tools. MERANT Micro Focus offers an embedded SQL interface for SQL Server directly
in its COBOL development environment.

Server Development Interface

SQL Server offers an open API for developing server-based applications that work in conjunction with SQL Server.
ODS is an event-driven API that is primarily used for developing extended stored procedures. ODS is actually a core
part of the SQL Server architecture and benefits from the high-performance architecture. It provides all the network,
connection, and thread management that SQL Server uses.

Formerly, ODS was frequently used to develop custom database gateways, data-driven event alerters, external
program triggers, and request auditing in addition to extended stored procedure DLLs. However, with the advent of
SQL Server distributed queries and newer technologies such as Windows Component Services, the need for ODS in
developing applications of this type of gateway has largely been eliminated. I strongly recommend that you investigate
other methods of developing the types of applications just mentioned and use ODS only for developing extended
stored procedures.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
The SQL Server component and product family—including the SQL Server RDBMS, visual systems management
tools, distributed systems components, open client/server interfaces, and visual development tools—provides a
complete and robust platform for developing and deploying large-scale database applications.

This chapter has provided an overview of the various tools available. The remainder of this book concentrates on the
capabilities and uses of the SQL Server 2000 engine, which is the foundation of the product.
This document is created with the unregistered version of CHM2PDF Pilot

34

Part II

Architectural Overview
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 3

SQL Server Architecture

Throughout this book, I'll be telling you about specific features of the Microsoft SQL Server engine. In this chapter,
I'll describe what makes up that engine and how it works. If you're interested only in the functional operations of SQL
Server and want to consider everything as a black box operation, you can safely skip this chapter. Or if you're new to
databases and don't have a clear idea of what makes up an RDBMS, you can come back to this chapter after you
become more familiar with SQL Server's external capabilities. This chapter will focus on what happens inside the
engine. I'll also point out system behavior that might affect application development and suggest ways to deal with it.
This document is created with the unregistered version of CHM2PDF Pilot

34

The SQL Server Engine

Figure 3-1 shows the general architecture of SQL Server. For simplicity, I've made some minor omissions and
simplifications and ignored certain "helper" modules.

Now let's look in detail at the major modules.

The Net-Library
The Net-Library (often called Net-Lib, but in this book I'll use Net-Library) abstraction layer enables SQL Server to
read from and write to many different network protocols, and each such protocol (such as TCP/IP sockets) can have
a specific driver. The Net-Library layer makes it relatively easy to support many different network protocols without
having to change the core server code.

A Net-Library is basically a driver that's specific to a particular network interprocess communication (IPC)
mechanism. (Be careful not to confuse driver with device driver.) All code in SQL Server, including Net-Library
code, makes calls only to the Microsoft Win32 subsystem. SQL Server uses a common internal interface between
Microsoft Open Data Services (ODS)—which manages its use of the network—and each Net-Library. If your
development project needs to support a new and different network protocol, you can handle all network-specific
issues by simply writing a new Net-Library. In addition, you can load multiple Net-Libraries simultaneously, one for
each network IPC mechanism in use.

SQL Server uses the Net-Library abstraction layer on both the server and client machines, making it possible to
support several clients simultaneously on different networks. Microsoft Windows NT/2000 and Windows 98 support
the simultaneous use of multiple protocol stacks. Net-Libraries are paired. For example, if a client application is using
a Named Pipes Net-Library, SQL Server must also be listening on a Named Pipes Net-Library. The client
application determines which Net-Library is actually used for the communication, and you can control the client
application's choice by using a tool called the Client Network Utility. You can easily configure SQL Server to listen
on multiple Net-Libraries by using the Server Network Utility, which is available under Programs\Microsoft SQL
Server on the Start menu.

SQL Server 2000 has two primary Net-Libraries: Super Socket and Shared Memory. TCP/IP, Named Pipes,
IPX/SPX, and so on are referred to as secondary Net-Libraries. The OLE DB Provider for SQL Server, SQL
Server ODBC driver, DB-Library, and the database engine communicate directly with these two primary network
libraries. Intercomputer connections communicate through the Super Socket Net-Library. Local connections between
an application and a SQL Server instance on the same computer use the Shared Memory Net-Library if Shared
Memory support has been enabled (which it is, by default). SQL Server 2000 supports the Shared Memory
Net-Library on all Windows platforms.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 3-1. The major components of the SQL Server architecture.

The Super Socket Net-Library has two components:

 Communication path If the client is configured to communicate over TCP/IP Sockets or NWLink
IPX/SPX connection, the Super Socket Net-Library directly calls the Windows Socket 2 API for the
communication between the application and the SQL Server instance.

If the client is configured to communicate over a Named Pipes, Multiprotocol, AppleTalk, or Banyan VINES
connection, a subcomponent of the Super Socket Net-Library called the Net-Library router loads the
secondary Net-Library for the chosen protocol and routes all Net-Library calls to it.
 Encryption layer The encryption is implemented using the Secure Sockets Layer (SSL) API. The level of
encryption, 40-bit or 128-bit, depends on the Windows version on the application and SQL Server-based
computers. Enabling encryption can slow network performance not only because of the extra work of
encrypting and decrypting communication between the client and the server but also because an extra
roundtrip is required between the client and the server every time a connection is made.

Shared Memory Net-Library communication is inherently secure without the need for encryption. The Shared
Memory Net-Library never participates in intercomputer communication. The area of memory shared
between the application process and the database engine process cannot be accessed from any other
Windows process.

For compatibility with earlier versions of SQL Server, the Multiprotocol Net-Library continues to support its own
encryption. This encryption is specified independently of the SSL encryption and is implemented by calling the
Windows RPC encryption API.

Figure 3-2 (taken from SQL Server Books Online) shows SQL Server 2000's Net-Library architecture.

The distinction between the IPC mechanisms and the underlying network protocols is important. IPC mechanisms
used by SQL Server include Named Pipes, RPC, SPX, and Windows Sockets. Network protocols used include
TCP/IP, NetBEUI, Shared Memory, NWLink IPX/SPX, Banyan VINES SPP, and AppleTalk ADSP. Two
Net-Libraries, Multiprotocol and Named Pipes, can be used simultaneously over multiple network protocols
(NetBEUI, NWLink IPX/SPX, and TCP/IP). You can have multiple network protocols in your environment and still
use only one Net-Library.

SQL Server 2000 running on Windows NT/2000 supports impersonation of security contexts to provide an
integrated logon authentication capability called Windows Authentication. Windows Authentication operates over
This document is created with the unregistered version of CHM2PDF Pilot

network protocols that support authenticated connections between clients and servers. Such connections are referred
to as trusted connections and are supported by SQL Server 2000 using any available Net-Library. Instead of
requiring a separate user ID/password logon each time a connection to SQL Server is requested, SQL Server can
impersonate the security context of the user running the application that requests the connection. If that user has
sufficient privileges (or is part of a Windows NT/2000 domain group that does), the connection is established. Note
that Windows Authentication is not available when SQL Server is running on Windows 98. When you connect to
SQL Server running on Windows 98, you must specify a SQL Server logon ID and password.

Figure 3-2. SQL Server 2000's Net-Library Architecture.

Which Net-Library Is Fastest?
Strictly speaking, the TCP/IP Sockets Net-Library is the fastest Net-Library. In a pure network test that does
nothing except throw packets back and forth between Net-Library pairs, the TCP/IP Sockets Net-Library is perhaps
30 percent faster than the slowest Net-Library. But in LAN environments and applications, the speed of the
Net-Library probably makes little difference because the network interface is generally not a limiting factor in a
well-designed application.

On a LAN, however, turning on encryption does cause a performance hit. But again, most applications probably
wouldn't notice the difference. Your best bet is to choose the Net-Library that matches your network protocols and
provides the services you need in terms of unified logon, encryption, and dynamic name resolution. (I'll explain these
choices further in Chapter 4.)

Open Data Services

Open Data Services (ODS) functions as the client manager for SQL Server; it's basically an interface between server
Net-Libraries and server-based applications, including SQL Server. ODS manages the network: it listens for new
connections, cleans up failed connections, acknowledges "attentions" (cancellations of commands), coordinates
threading services to SQL Server, and returns result sets, messages, and status values back to the client.
This document is created with the unregistered version of CHM2PDF Pilot

SQL Server clients and the server speak a private protocol known as Tabular Data Stream (TDS). TDS is a
self-describing data stream. In other words, it contains tokens that describe column names, datatypes, events (such as
cancellations), and status values in the "conversation" between client and server. The server notifies the client that it is
sending a result set, indicates the number of columns and datatypes of the result set, and so on—all encoded in TDS.
Neither clients nor servers write directly to TDS. Instead, the open interfaces of OLE-DB, ODBC, and DB-Library
at the client emit TDS using a client implementation of the Net-Library.

ODS accepts new connections, and if a client unexpectedly disconnects (for example, if a user reboots the client
computer instead of cleanly terminating the application), ODS automatically frees resources such as locks held by that
client.

You can use the ODS open interface to extend the functionality of SQL Server by writing extended stored
procedures. The code for the ODS API itself was formerly stored outside of SQL Server as part of the DLL
opends60.dll. In SQL Server 2000, to enhance the performance of SQL Server's own internal mechanisms that need
to use presupplied extended procedures, the code that makes up ODS is internal to the SQL Server engine. Any
extended stored procedure DLLs that you create must link against opends60.dll, which contains stubs to access the
real ODS routines internal to SQL Server.

ODS input and output buffers

After SQL Server puts result sets into a network output buffer (write buffer) that's equal in size to the configured
packet size, the Net-Library dispatches the buffer to the client. The first packet is sent as soon as the network output
buffer is full or, if an entire result set fits in one packet, when the batch is completed. (A batch is one or more
commands sent to SQL Server to be parsed and executed together. For example, if you're using OSQL.EXE, a
batch is the collection of all the commands that appear before a specific GO command.) In some exceptional
operations (such as one that provides progress information for database dumping or provides DBCC messages), the
output buffer is flushed and sent even before it is full or before the batch completes.

SQL Server has two input buffers (read buffers) and one output buffer per client. Double-buffering is needed for the
reads because while SQL Server reads a stream of data from the client connection, it must also look for a possible
attention. (This allows that "Query That Ate Cleveland" to be canceled directly from the issuer. Although the ability to
cancel a request is extremely important, it's relatively unusual among client/server products.) Attentions can be thought
of as "out-of-band" data, although they can be sent with network protocols that do not explicitly have an out-of-band
channel. The SQL Server development team experimented with double-buffering and asynchronous techniques for the
output buffers, but these didn't improve performance substantially. The single network output buffer works nicely.
Even though the writes are not posted asynchronously, SQL Server doesn't need to bypass the operating system
caching for these as it does for writes to disk.

Because the operating system provides caching of network writes, write operations appear to complete immediately
with no significant latency. If, however, several writes are issued to the same client and the client is not currently
reading data from the network, the network cache eventually becomes full and the write blocks. This is essentially a
throttle. As long as the client application is processing results, SQL Server has a few buffers queued up and ready for
the client connection to process. But if the client's queue is already stacked up with results and is not processing them,
SQL Server stalls sending them and the network write operation to that connection has to wait. Since the server has
only one output buffer per client, data cannot be sent to that client connection until it reads information off the network
to free up room for the write to complete. (Writes to other client connections are not held up, however; only those for
the laggard client are affected.)

Stalled network writes can also affect locks. For example, if READ COMMITTED isolation is in effect (the default),
a share lock can normally be released after SQL Server has completed its scan of that page of data. (Exclusive locks
used for changing data must always be held until the end of the transaction to ensure that the changes can be rolled
back.) However, if the scan finds more qualifying data and the output buffer is not free, the scan stalls. When the
previous network write completes, the output buffer becomes available and the scan resumes. But as I stated earlier,
that write won't complete until the client connection "drains" (reads) some data to free up some room in the pipe (the
This document is created with the unregistered version of CHM2PDF Pilot

virtual circuit between SQL Server and client connection).

If a client connection delays processing results that are sent to it, concurrency issues can result because locks are held
longer than they otherwise would be. A kind of chain reaction occurs: if the client connection has not read several
outstanding network packets, further writing of the output buffer at the SQL Server side must wait because the pipe is
full. Since the output buffer is not available, the scan for data might also be suspended because no space is available
to add qualifying rows. Since the scan is held up, any lock on the data cannot be released. In short, if a client
application does not process results in a timely manner, database concurrency can suffer.

The size of the network output buffer can also affect the speed at which the client receives the first result set. As
mentioned earlier, the output buffer is sent when the batch, not simply the command, is done, even if the buffer is not
full. If two queries exist in the same batch and the first query has only a small amount of data, its results are not sent
back to the client until the second query is done or has supplied enough data to fill the output buffer. If both queries
are fast, waiting for both queries to finish is not a problem. But suppose the first query is fast and the second is slow.
And suppose the first query returns 1000 bytes of data. If the network packet size is 4096 bytes, the first result set
must wait in the output buffer for the second query to fill it. The obvious solution here is to either make the first
command its own batch or make the network packet size smaller. The first solution is probably best in this case, since
it is typically difficult to fine-tune your application to determine the best buffer size for each command. But this doesn't
mean that each command should be its own batch. Quite the contrary. In fact, under normal circumstances, grouping
multiple commands into a single batch is most efficient and is recommended because it reduces the amount of
handshaking that must occur between client and server.

ODS default Net-Libraries

On the server side, ODS provides functionality that mirrors that of ODBC, OLE DB, or DB-Library at the client.
Calls exist for an ODS server application to describe and send result sets, to convert values between datatypes, to
assume the security context associated with the specific connection being managed, and to raise errors and messages
to the client application.

ODS uses an event-driven programming model. Requests from servers and clients trigger events that your server
application must respond to. Using the ODS API, you create a custom routine, called an event handler, for each
possible type of event. Essentially, the ODS library drives a server application by calling its custom event handlers in
response to incoming requests.

ODS server applications respond to the following events:

 Connect events When a connect event occurs, SQL Server initiates a security check to determine whether
a connection is allowed. Other ODS applications, such as a gateway to DB/2, have their own logon handlers
that determine whether connections are allowed. Events also exist that close a connection, allowing the proper
connection cleanup to occur.
 Language events When a client sends a command string, such as an SQL statement, SQL Server passes
this command along to the command parser. A different ODS application, such as a gateway, would install its
own handler that accepts and is responsible for execution of the command.
 Remote stored procedure events These events occur each time a client or SQL Server directs a remote
stored procedure request to ODS for processing.

ODS also generates events based on certain client activities and application activities. These events allow an ODS
server application to respond to changes to the status of the client connection or of the ODS server application.

In addition to handling connections, ODS manages threads (and fibers) for SQL Server. It takes care of thread
creation and termination and makes the threads available to the User Mode Scheduler (UMS). Since ODS is an open
interface with a full programming API and toolkit, independent software vendors (ISVs) writing server applications
with ODS get the same benefits that SQL Server derives from this component, including SMP-capable thread
management and pooling, as well as network handling for multiple simultaneous networks. This multithreaded
operation enables ODS server applications to maintain a high level of performance and availability and to
This document is created with the unregistered version of CHM2PDF Pilot

transparently use multiple processors under Windows NT/2000 because the operating system can schedule any
thread on any available processor.

The Relational Engine and the Storage Engine

The SQL Server database engine is made up of two main components, the relational engine and the storage engine.
These two pieces are clearly separated, and their primary method of communication with each other is through OLE
DB. The relational engine comprises all the components necessary to parse and optimize any query. It also manages
the execution of queries as it requests data from the storage engine in terms of OLE DB row sets and then processes
the row sets returned. (Row set is the OLE DB term for a result set.) The storage engine comprises the components
needed to actually access and modify data on disk.

The command parser

The command parser handles language events raised by ODS. It checks for proper syntax and translates
Transact-SQL commands into an internal format that can be operated on. This internal format is known as a query
tree. If the parser doesn't recognize the syntax, a syntax error is immediately raised and identifies where the error
occurred. However, non-syntax error messages cannot be explicit about the exact source line that caused the error.
Because only the command parser can access the source of the statement, the statement is no longer available in
source format when the command is actually executed.

The optimizer
The optimizer takes the query tree from the command parser and prepares it for execution. This module compiles an
entire command batch, optimizes queries, and checks security. The query optimization and compilation result in an
execution plan.

The first step in producing such a plan is to normalize each query, which potentially breaks down a single query into
multiple, fine-grained queries. After the optimizer normalizes a query, it optimizes it, which means that the optimizer
determines a plan for executing that query. Query optimization is cost-based; the optimizer chooses the plan that it
determines would cost the least based on internal metrics that include estimated memory requirements, estimated
CPU utilization, and the estimated number of required I/Os. The optimizer considers the type of statement requested,
checks the amount of data in the various tables affected, looks at the indexes available for each table, and then looks
at a sampling of the data values kept for each index or column referenced in the query. The sampling of the data
values is called distribution statistics and will be discussed in detail in Chapter 15. Based on the available
information, the optimizer considers the various access methods and join strategies it could use to resolve a query and
chooses the most cost-effective plan. The optimizer also decides which indexes, if any, should be used for each table
in the query, and, in the case of a multitable query, the order in which the tables should be accessed and the join
strategy to use.

The optimizer also uses pruning heuristics to ensure that more time isn't spent optimizing a query than it would take to
simply choose a plan and execute it. The optimizer doesn't necessarily do exhaustive optimization. Some products
consider every possible plan and then choose the most cost-effective one. The advantage of this exhaustive
optimization is that the syntax chosen for a query would theoretically never cause a performance difference, no matter
what syntax the user employed. But if you deal with an involved query, it could take much longer to estimate the cost
of every conceivable plan than it would to accept a good plan, even if not the best one, and execute it.

After normalization and optimization are completed, the normalized tree produced by those processes is compiled
into the execution plan, which is actually a data structure. Each command included in it specifies exactly which table
will be affected, which indexes will be used (if any), which security checks must be made, and which criteria (such as
equality to a specified value) must evaluate to TRUE for selection. This execution plan might be considerably more
complex than is immediately apparent. In addition to the actual commands, the execution plan includes all the steps
necessary to ensure that constraints are checked. Steps for calling a trigger are a bit different from those for verifying
This document is created with the unregistered version of CHM2PDF Pilot

constraints. If a trigger is included for the action being taken, a call to the procedure that comprises the trigger is
appended. If the trigger is an instead-of trigger, the call to the trigger's plan replaces the actual data modification
command. For after triggers, the trigger's plan is branched to right after the plan for the modification statement that
fired the trigger, before that modification is committed. The specific steps for the trigger are not compiled into the
execution plan, unlike those for constraint verification.

A simple request to insert one row into a table with multiple constraints can result in an execution plan that requires
many other tables to also be accessed or expressions to be evaluated. The existence of a trigger can also cause many
additional steps to be executed. The step that carries out the actual INSERT statement might be just a small part of
the total execution plan necessary to ensure that all actions and constraints associated with adding a row are carried
out.

For more details on query processing and optimization, see Chapter 15.

The SQL manager

The SQL manager is responsible for everything having to do with managing stored procedures and their plans. It
determines when a stored procedure needs recompilation based on changes in the underlying objects' schemas, and it
manages the caching of procedure plans so that other processes can reuse them.

The SQL manager also handles autoparameterization of queries. In SQL Server 2000, certain kinds of ad hoc
queries are treated as if they were parameterized stored procedures, and query plans are generated and saved for
them. This can happen if a query uses a simple equality comparison against a constant, as in the following statement:
SELECT * FROM pubs.dbo.titles
WHERE type = 'business'

This query can be parameterized as if it were a stored procedure with a parameter for the value of type:
SELECT * FROM pubs.dbo.titles
WHERE type = @param

A subsequent query, differing only in the actual value used for the value of type, can use the same query plan that was
generated for the original query. I'll discuss the details of autoparameterization in Chapter 15.

The expression manager

The expression manager handles computation, comparison, and data movement. Suppose your query contains an
expression like this one:
SELECT @myqty = qty * 10 FROM mytable

The expression manager copies the value of qty from the row set returned by the storage engine, multiplies it by 10,
and stores the result in @myqty.

The query executor

The query executor runs the execution plan that the optimizer produced, acting as a dispatcher for all the commands
in the execution plan. This module loops through each command step of the execution plan until the batch is complete.
Most of the commands require interaction with the storage engine to modify or retrieve data and to manage
transactions and locking.
This document is created with the unregistered version of CHM2PDF Pilot

Communication between the relational engine and the storage

engine
The relational engine uses OLE DB for most of its communication with the storage engine. The following description
of that communication is adapted from the section titled "Database Engine Components" in SQL Server Books
Online. It describes how a SELECT statement that processes data from local tables only is processed:
1. The relational engine compiles the SELECT statement into an optimized execution plan. The execution plan
defines a series of operations against simple OLE DB row sets from the individual tables or indexes
referenced in the SELECT statement. The row sets requested by the relational engine return the amount of
data needed from a table or index to perform one of the operations used to build the SELECT result set. For
example, this SELECT statement requires a table scan if it references a table with no indexes:
SELECT * FROM Northwind.dbo.ScanTable

The relational engine implements the table scan by requesting one row set containing all the rows from
ScanTable. This next SELECT statement needs only information available in an index:
SELECT DISTINCT LastName
FROM Northwind.dbo.Employees

The relational engine implements the index scan by requesting one row set containing the leaf rows from the
index that was built on the LastName column. The following SELECT statement needs information from two
indexes:
SELECT CompanyName, OrderID, ShippedDate
FROM Northwind.dbo.Customers AS Cst
JOIN Northwind.dbo.Orders AS Ord
ON (Cst.CustomerID = Ord.CustomerID)

The relational engine requests two row sets: one for the nonclustered index on Customers and the other for
one of the clustered indexes on Orders.
2. The relational engine uses the OLE DB API to request that the storage engine open the row sets.
3. As the relational engine works through the steps of the execution plan and needs data, it uses OLE DB to
fetch the individual rows from the row sets it asked the storage engine to open. The storage engine transfers
the data from the data buffers to the relational engine.
4. The relational engine combines the data from the storage engine row sets into the final result set transmitted
back to the user.

Not all communication between the relational engine and the storage engine uses OLE DB. Some commands cannot
be expressed in terms of OLE DB row sets. The most obvious and common example is when the relational engine
processes data definition language (DDL) requests to create a table or other SQL Server object.

The Access Methods Manager

When SQL Server needs to locate data, it calls the access methods manager. The access methods manager sets up
and requests scans of data pages and index pages and prepares the OLE DB row sets to return to the relational
engine. It contains services to open a table, retrieve qualified data, and update data. The access methods manager
doesn't actually retrieve the pages; it makes the request to the buffer manager, which ultimately serves up the page
already in its cache or reads it to cache from disk. When the scan is started, a look-ahead mechanism qualifies the
rows or index entries on a page. The retrieving of rows that meet specified criteria is known as a qualified retrieval.
The access methods manager is employed not only for queries (selects) but also for qualified updates and deletes (for
example, UPDATE with a WHERE clause).
This document is created with the unregistered version of CHM2PDF Pilot

A session opens a table, requests and evaluates a range of rows against the conditions in the WHERE clause, and
then closes the table. A session descriptor data structure (SDES) keeps track of the current row and the search
conditions for the object being operated on (which is identified by the object descriptor data structure, or DES).

The Row Operations Manager and the Index

Manager
You can consider the row operations manager and the index manager as components of the access methods manager
because they carry out the actual method of access. Each is responsible for manipulating and maintaining its respective
on-disk data structures, namely rows of data or B-tree indexes. They understand and manipulate information on data
and index pages.

The row operations manager

The row operations manager retrieves, modifies, and performs operations on individual rows. (For more information
about on-disk structures, see Chapter 5; for more on data page and row format, see Chapter 6.) It performs an
operation within a row, such as "retrieve column 2" or "write this value to column 3." As a result of the work
performed by the access methods manager, as well as by the lock manager and transaction manager, which will be
discussed shortly, the row will have been found and will be appropriately locked and part of a transaction. After
formatting or modifying a row in memory, the row operations manager inserts or deletes a row.

The row operations manager also handles updates. SQL Server 2000 offers three methods for handling updates. All
three are direct, which means that there's no need for two passes through the transaction log, as was the case with
deferred updates in versions of SQL Server prior to SQL Server 7. SQL Server 2000 has no concept of a deferred
data modification operation.

SQL Server 2000 has three update modes:

 In-place mode This mode is used to update a heap or clustered index when none of the clustering keys
change. The update can be done in place, and the new data is written to the same slot on the data page.
 Split mode This mode is used to update nonunique indexes when the index keys change. The update is split
into two operations—a delete followed by an insert—and these operations are performed independently of
each other.
 Split with collapse mode This mode is used to update a unique index when the index keys change. After the
update is rewritten as delete and insert operations, if the same index key is both deleted and then reinserted
with a new value, the delete and insert are "collapsed" into a single update operation.

In Chapter 9, we'll look at examples of each type of update.

The index manager

The index manager maintains and supports searches on B-trees, which are used for SQL Server indexes. An index is
structured as a tree, with a root page and intermediate-level and lower-level pages (or branches). A B-tree groups
records that have similar index keys, thereby allowing fast access to data by searching on a key value. The B-tree's
core feature is its ability to balance the index tree. (B stands for balanced.) Branches of the index tree are spliced
together or split apart as necessary so that the search for any given record always traverses the same number of levels
and thus requires the same number of page accesses.

The traversal begins at the root page, progresses to intermediate index levels, and finally moves to bottom-level
pages called leaf pages. The index is used to find the correct leaf page. On a qualified retrieval or delete, the correct
leaf page is the lowest page of the tree at which one or more rows with the specified key or keys reside. SQL Server
supports both clustered and nonclustered indexes. In a nonclustered index, shown in Figure 3-3, the leaf level of the
This document is created with the unregistered version of CHM2PDF Pilot

tree (the leaf pages of the index) contains every key value in the index along with a bookmark for each key value. The
bookmark indicates where to find the referenced data and can have one of two forms, depending on whether the base
table has a clustered index. If the base table has no clustered index, the table is referred to as a heap. The bookmarks
in nonclustered index leaf pages for a heap are pointers to the actual records in which the data can be found, and
these pointers consist of a row ID (RID), which is a file number, a page number, and a row number on the page. If
the base table has a clustered index, the bookmark in any nonclustered index leaf page contains the clustered index
key value for the row.

After reaching the leaf level in a nonclustered index, you can find the exact location of the data, although you still must
separately retrieve the page on which that data resides. Because you can access the data directly, you don't need to
scan all the data pages to find a qualifying row. Better yet, in a clustered index, shown in Figure 3-4, the leaf level
actually contains the full data rows, not simply the index keys. A clustered index keeps the data in a table logically
ordered around the key of the clustered index, and the leaf pages of a clustered index are in fact the data pages of the
table. All the data pages of a table with a clustered index are linked together in a doubly linked list. Following the
pages, and the rows on those pages, from the first page to the last page provides the logical order to the data.

Because data can be ordered in only one way, only one clustered index can exist per table. This makes the selection
of the appropriate key value on which to cluster data an important performance consideration.

You can also use indexes to ensure the uniqueness of a particular key value. In fact, the PRIMARY KEY and
UNIQUE constraints on a column work by creating a unique index on the column's values. The optimizer can use the
knowledge that an index is unique in formulating an effective query plan. Internally, SQL Server always ensures that
clustered indexes are unique by adding a 4-byte uniqueifier to clustered index key values that occur more than once.
This uniqueifier becomes part of the key and is used in all levels of the clustered index and in references to the
clustered index key through all nonclustered indexes.

Figure 3-3. A nonclustered index with the leaf level containing bookmarks—either a RID or a clustered index
key value.

Since SQL Server maintains ordering in index leaf levels, you do not need to unload and reload data to maintain
clustering properties as data is added and moved. SQL Server always inserts rows into the correct page in clustered
sequence. For a clustered index, the correct leaf page is the data page in which a row is inserted. For a nonclustered
index, the correct leaf page is the one into which SQL Server inserts a row containing the key value (and bookmark)
for the newly inserted row. If data is updated and the key values of an index change, or if the row is moved to a
different page, SQL Server's transaction control ensures that all affected indexes are modified to reflect these
This document is created with the unregistered version of CHM2PDF Pilot

changes. With transaction control, index operations are performed as atomic operations. The operations are logged
and fully recovered in the event of a system failure.

For more details on index structures and maintenance, see Chapter 7.

Figure 3-4. A clustered index with the data located at the leaf level.
Locking and Index Pages
As you'll see later, in the section on the lock manager, pages of an index use a slightly different locking mechanism
than regular data pages. A lightweight lock called a latch is used to lock upper levels of indexes. Latches are not
involved in deadlock detection because SQL Server 2000 uses "deadlock-proof" algorithms for index maintenance.

You can customize the locking strategy for indexes on a table basis or index basis. You can use the system stored
procedure sp_indexoption to enable or disable page or row locks with any particular index or, by specifying a table
name, for every index on that table. The settable options are DisAllowPageLocks and DisAllowRowLocks. If both
of these options are set to TRUE for a particular index, only table level locks are applied.

The Page Manager and the Text Manager

The page manager and the text manager cooperate to manage a collection of pages as named databases. Each
database is a collection of 8-KB disk pages, which are spread across one or more physical files. (In Chapter 5, you'll
find more details about the physical organization of databases.)

SQL Server uses eight types of disk pages: data pages, text/image pages, index pages, Page Free Space (PFS)
pages, Global Allocation Map (GAM and SGAM) pages, Index Allocation Map (IAM) pages, Bulk Changed Map
pages, and Differential Changed Map pages. All user data, except for the text, ntext, and image datatypes, are
stored on data pages. These three datatypes, which are used for storing large objects (up to 2 GB each of text or
binary data), can use a separate collection of pages, so the data is not typically stored on regular data pages with the
rest of the rows. Instead, a pointer on the regular data page identifies the starting page and offset of the text/image
data. However, in SQL Server 2000, large object data that contains only a few bytes can optionally be stored in the
data row itself. Index pages store the Btrees that allow fast access to data. PFS pages keep track of which pages in a
database are available to hold new data. Allocation pages (GAMs, SGAMs, and IAMs) keep track of the other
pages. They contain no database rows and are used only internally. Bulk Changed Map pages and Differential
Changed Map pages are used to make backup and recovery more efficient. I'll cover all the types of pages
extensively in Chapters 5, 6, and 8.

The page manager allocates and deallocates all types of disk pages, organizing extents of eight pages each. An extent
can be either a uniform extent, for which all eight pages are allocated to the same object (table or index), or a mixed
extent, which can contain pages from multiple objects. If an object uses fewer than eight pages, the page manager
This document is created with the unregistered version of CHM2PDF Pilot

allocates new pages for that object from mixed extents. When the size of the object exceeds eight pages, the page
manager allocates new space for that object in units of entire uniform extents. This optimization prevents the overhead
of allocation from being incurred every time a new page is required for a large table; this overhead is incurred only
every eighth time. Perhaps most important, this optimization forces data of the same table to be contiguous, for the
most part. At the same time, the ability to use mixed extents keeps SQL Server from wasting too much space if a
database contains many small tables.

To determine how contiguous a table's data is, you use the DBCC SHOWCONTIG command. A table with a lot of
allocation and deallocation can get fairly fragmented, and rebuilding the clustered index (which also rebuilds the table)
or running DBCC INDEXDEFRAG can improve performance, especially when a table is accessed frequently using
ordered index scans. I'll discuss these two commands in detail in Chapter 8.

The Transaction Manager

A core feature of SQL Server is its ability to ensure that transactions follow the ACID properties (discussed in
Chapter 2). Transactions must be atomic—that is, all or nothing. If a transaction has been committed, it must be
recoverable by SQL Server no matter what—even if a total system failure occurs one millisecond after the commit
was acknowledged. In SQL Server, if work was in progress and a system failure occurred before the transaction was
committed, all the work is rolled back to the state that existed before the transaction began. Write-ahead logging
makes it possible to always roll back work in progress or roll forward committed work that has not yet been applied
to the data pages. Write-ahead logging ensures that a transaction's changes—the "before and after" images of
data—are captured on disk in the transaction log before a transaction is acknowledged as committed. Writes to the
transaction log are always synchronous—that is, SQL Server must wait for them to complete. Writes to the data
pages can be asynchronous because all the effects can be reconstructed from the log if necessary. The transaction
manager coordinates logging, recovery, and buffer management. These topics are discussed later in this chapter; at
this point, we'll just look at transactions themselves.

The transaction manager delineates the boundaries of statements that must be grouped together to form an operation.
It handles transactions that cross databases within the same SQL Server, and it allows nested transaction sequences.
(However, nested transactions simply execute in the context of the first-level transaction; no special action occurs
when they are committed. And a rollback specified in a lower level of a nested transaction undoes the entire
transaction.) For a distributed transaction to another SQL Server (or to any other resource manager), the transaction
manager coordinates with the Microsoft Distributed Transaction Coordinator (MS DTC) service using operating
system remote procedure calls. The transaction manager marks savepoints, which let you designate points within a
transaction at which work can be partially rolled back or undone.

The transaction manager also coordinates with the lock manager regarding when locks can be released, based on the
isolation level in effect. The isolation level in which your transaction runs determines how sensitive your application is
to changes made by others and consequently how long your transaction must hold locks to protect against those
changes. Four isolation-level semantics are available in SQL Server 2000: Uncommitted Read (also called "dirty
read"), Committed Read, Repeatable Read, and Serializable.

The behavior of your transactions depends on the isolation level. We'll look at these levels now, but a complete
understanding of isolation levels also requires an understanding of locking because the topics are so closely related.
The next section gives an overview of locking; you'll find more detailed information in Chapter 14.

Uncommitted Read
Uncommitted Read, or dirty read (not to be confused with "dirty page," which I'll discuss later) lets your transaction
read any data that is currently on a data page, whether or not that data has been committed. For example, another
user might have a transaction in progress that has updated data, and even though it's holding exclusive locks on the
data, your transaction can read it anyway. The other user might then decide to roll back his or her transaction, so
logically those changes were never made. If the system is a single-user system and everyone is queued up to access it,
This document is created with the unregistered version of CHM2PDF Pilot

the changes will not have been visible to other users. In a multiuser system, however, you read the changes and take
action based on them. Although this scenario isn't desirable, with Uncommitted Read you can't get stuck waiting for a
lock, nor do your reads issue share locks (described in the next section) that might affect others.

When using Uncommitted Read, you give up the assurance of strongly consistent data in favor of high concurrency in
the system without users locking each other out. So when should you choose Uncommitted Read? Clearly, you don't
want to use it for financial transactions in which every number must balance. But it might be fine for certain
decision-support analyses—for example, when you look at sales trends—for which complete precision isn't necessary
and the tradeoff in higher concurrency makes it worthwhile.

Committed Read
Committed Read is SQL Server's default isolation level. It ensures that an operation never reads data that another
application has changed but not yet committed. (That is, it never reads data that logically never existed.) With
Committed Read, if a transaction is updating data and consequently has exclusive locks on data rows, your
transaction must wait for those locks to be released before you can use that data (whether you're reading or
modifying). Also, your transaction must put share locks (at a minimum) on the data that will be visited, which means
that data might be unavailable to others to use. A share lock doesn't prevent others from reading the data, but it
makes them wait to update the data. Share locks can be released after the data has been sent to the calling
client—they don't have to be held for the duration of the transaction.
NOTE
Although a transaction can never read uncommitted data when running with Committed Read isolation, if the
transaction subsequently revisits the same data, that data might have changed or new rows might suddenly appear that
meet the criteria of the original query. If data values have changed, we call that a non-repeatable read. New rows
that appear are called phantoms.

Repeatable Read
The Repeatable Read isolation level adds to the properties of Committed Read by ensuring that if a transaction
revisits data or if a query is reissued, the data will not have changed. In other words, issuing the same query twice
within a transaction will not pick up any changes to data values made by another user's transaction. However,
Repeatable Read isolation level does allow phantom rows to appear.

Preventing nonrepeatable reads from appearing is a desirable safeguard. But there's no free lunch. The cost of this
extra safeguard is that all the shared locks in a transaction must be held until the completion (COMMIT or
ROLLBACK) of the transaction. (Exclusive locks must always be held until the end of a transaction, no matter what
the isolation level, so that a transaction can be rolled back if necessary. If the locks were released sooner, it might be
impossible to undo the work.) No other user can modify the data visited by your transaction as long as your
transaction is outstanding. Obviously, this can seriously reduce concurrency and degrade performance. If transactions
are not kept short or if applications are not written to be aware of such potential lock contention issues, SQL Server
can appear to "hang" when it's simply waiting for locks to be released.
NOTE
You can control how long SQL Server waits for a lock to be released by using the session option
LOCK_TIMEOUT. I'll discuss this option, as well as many other ways of customizing locking behavior, in Chapter
16.

Serializable
The Serializable isolation level adds to the properties of Repeatable Read by ensuring that if a query is reissued, rows
will not have been added in the interim. In other words, phantoms will not appear if the same query is issued twice
within a transaction. More precisely, Repeatable Read and Serializable affect sensitivity to another connection's
changes, whether or not the user ID of the other connection is the same. Every connection within SQL Server has its
This document is created with the unregistered version of CHM2PDF Pilot

own transaction and lock space. I use the term "user" loosely so as not to obscure the central concept.

Preventing phantoms from appearing is another desirable safeguard. But once again, there's no free lunch. The cost of
this extra safeguard is similar to that of Repeatable Read—all the shared locks in a transaction must be held until
completion of the transaction. In addition, enforcing the Serializable isolation level requires that you not only lock data
that has been read, but also lock data that does not exist! For example, suppose that within a transaction we issue a
SELECT statement to read all the customers whose zip code is between 98000 and 98100, and on first execution no
rows satisfy that condition. To enforce the Serializable isolation level, we must lock that "range" of potential rows
with zip codes between 98000 and 98100 so that if the same query is reissued, there will still be no rows that satisfy
the condition. SQL Server handles this by using a special kind of lock called a key-range lock, which I'll discuss in
Chapter 14. The Serializable level gets its name from the fact that running multiple serializable transactions at the same
time is the equivalent of running them one at a time—that is, serially. For example, suppose transactions A, B, and C
run simultaneously at the Serializable level and each tries to update the same range of data. If the order in which the
transactions acquire locks on the range of data is B, C, and A, the result obtained by running all three simultaneously
is the same as if they were run one at a time in the order B, C, and A. Serializable does not imply that the order is
known in advance. The order is considered a chance event. Even on a single-user system, the order of transactions
hitting the queue would be essentially random. If the batch order is important to your application, you should
implement it as a pure batch system.
SEE ALSO
For an interesting critique of the formal definitions of the ANSI isolation levels, see the companion CD, which
contains a technical report called "A Critique of ANSI SQL Isolation Levels," published by the Microsoft Research
Center.

The tough part of transaction management, of course, is dealing with rollback/rollforward and recovery operations. I'll
return to the topic of transaction management and recovery a bit later. But first I'll discuss locking and logging further.

The Lock Manager

Locking is a crucial function of a multiuser database system such as SQL Server. Recall from Chapter 2 that SQL
Server lets you manage multiple users simultaneously and ensures that the transactions observe the properties of the
chosen isolation level. At the highest level, Serializable, SQL Server must make the multiuser system perform like a
single-user system—as though every user is queued up to use the system alone with no other user activity. Locking
guards data and the internal resources that make it possible for many users to simultaneously access the database and
not be severely affected by others' use.

The lock manager acquires and releases various types of locks, such as share locks for reading, exclusive locks for
writing, intent locks to signal a potential "plan" to perform some operation, extent locks for space allocation, and so
on. It manages compatibility between the lock types, resolves deadlocks, and escalates locks if needed. The lock
manager controls table, page, and row locks as well as system data locks. (System data, such as page headers and
indexes, are private to the database system.)

The lock manager provides two separate locking systems. The first enables row locks, page locks, and table locks
for all fully shared data tables, data pages and rows, text pages, and leaf-level index pages and index rows. The
second locking system is used internally only for restricted system data; it protects root and intermediate index pages
while indexes are being traversed. This internal mechanism uses latches, a lightweight, short-term variation of a lock
for protecting data that does not need to be locked for the duration of a transaction. Full-blown locks would slow the
system down. In addition to protecting upper levels of indexes, latches protect rows while they are being transferred
from the storage engine to the relational engine. If you examine locks by using the sp_lock stored procedure or a
similar mechanism that gets its information from the syslockinfo system table, you won't see or be aware of latches;
you'll see only the locks for fully shared data. However, counters are available in the System Monitor to monitor latch
requests, acquisitions, and releases. These will be discussed in Chapter 17.

Locking is an important aspect of SQL Server. Many developers are keenly interested in locking because of its
This document is created with the unregistered version of CHM2PDF Pilot

potential effect on application performance. Chapter 14 is devoted to the subject, so I won't go into it further here.

Other Managers
Also included in the storage engine are managers for controlling utilities such as bulk load, DBCC commands,
backup and restore operations, and the Virtual Device Interface (VDI). VDI allows ISVs to write their own backup
and restore utilities and to access the SQL Server data structures directly, without going through the relational engine.
There is a manager to control sorting operations and one to physically manage the files and backup devices on disk.
This document is created with the unregistered version of CHM2PDF Pilot

34

Managing Memory
One of the major goals of SQL Server 2000 was to scale easily from a laptop installation on Windows 98 to an
SMP server running on Windows 2000 DataCenter Server. This ability to scale requires a robust policy for managing
memory. By default, SQL Server 2000 adjusts its use of system memory to balance the needs of other applications
running on the machine and the needs of its own internal components. These other applications can include other
instances of SQL Server running on the same machine.

When determining the optimal amount of memory to dynamically allocate to SQL Server, the memory manager
strives to keep at least 4 MB of memory completely free and available for the operating system, but it might try to
keep more available. The amount of free space reserved for the operating system is partially determined by the
average life expectancy of a page in cache. The memory manager determines a Life Expectancy value that indicates
how many seconds, on average, a page stays in cache if it isn't referenced at all. Performance Monitor has a counter
that allows you to view the current Life Expectancy value. That value is of course only an estimate, and it's based on
the speed at which the lazywriter checks for referenced pages and the total amount of memory available. (I'll talk
about the lazywriter later in this section.) If the average Life Expectancy of a page is small, the memory manager tries
to keep the reserve memory at about 4 MB to increase the amount of memory available to SQL Server. If the
average Life Expectancy increases, the space reserved for the operating system increases to up to 10 MB and the
total memory for SQL Server decreases. All instances of SQL Server on a single machine tend to work toward the
same reserve value and a similar Life Expectancy value.

As an alternative to dynamically computing the total amount of memory, you can configure SQL Server to use a fixed
amount of memory. Whether memory allocation is fixed or dynamically adjusted, the total memory space for each
instance is considered one unified cache and is managed as a collection of various pools with their own policies and
purposes. Memory can be requested by and granted to any of several internal components.

The Buffer Manager and Memory Pools

The main memory component in SQL Server is the buffer pool; all memory not used by another memory component
remains in the buffer pool. The buffer manager manages disk I/O functions for bringing data and index pages into the
buffer pool so that data can be shared among users. When other components require memory, they can request a
buffer from the buffer pool. A buffer is a page in memory that's the same size as a data or index page. You can think
of it as a page frame that can hold one page from a database.

Another memory pool is the operating system itself. Occasionally, SQL Server must request contiguous memory in
larger blocks than the 8-KB pages that the buffer pool can provide. Typically, use of large memory blocks is kept to
a minimum, so direct calls to the operating system account for a small fraction of SQL Server's memory usage.

You can think of the procedure cache as another memory pool, in which query trees and plans from stored
procedures, triggers, user-defined functions or ad hoc queries can be stored. Other pools are used by
memory-intensive queries that use sorting or hashing, and by special memory objects that need less than one 8-KB
page.

Access to In-Memory Pages

Access to pages in the buffer pool must be fast. Even with real memory, it would be ridiculously inefficient to have to
scan the whole cache for a page when you have hundreds of megabytes, or even gigabytes, of data. To avoid this
inefficiency, pages in the buffer pool are hashed for fast access. Hashing is a technique that uniformly maps a key via
This document is created with the unregistered version of CHM2PDF Pilot

a hash function across a set of hash buckets. A hash bucket is a structure in memory that contains an array of
pointers (implemented as a linked list) to the buffer pages. If all the pointers to buffer pages do not fit on a single hash
page, a linked list chains to additional hash pages.

Given a dbid-fileno-pageno identifier (a combination of the database ID, file number, and page number), the hash
function converts that key to the hash bucket that should be checked; in essence, the hash bucket serves as an index
to the specific page needed. By using hashing, even when large amounts of memory are present, SQL Server can find
a specific data page in cache with only a few memory reads. Similarly, it takes only a few memory reads for SQL
Server to determine that a desired page is not in cache and that it must be read in from disk.
NOTE
Finding a data page might require that multiple hash buckets be accessed via the chain (linked list). The hash function
attempts to uniformly distribute the dbid-fileno-pageno values throughout the available hash buckets. The number of
hash buckets is set internally by SQL Server and depends on the total size of the buffer pool.

Access to Free Pages (Lazywriter)

You can use a data page or an index page only if it exists in memory. Therefore, a buffer in the buffer pool must be
available for the page to be read into. Keeping a supply of buffers available for immediate use is an important
performance optimization. If a buffer isn't readily available, many memory pages might have to be searched simply to
locate a buffer to free up for use as a workspace.

The buffer pool is managed by a process called the lazywriter that uses a clock algorithm to sweep through the buffer
pool. Basically, a lazywriter thread maintains a pointer into the buffer pool that "sweeps" sequentially through it (like
the hand on a clock). As the lazywriter visits each buffer, it determines whether that buffer has been referenced since
the last sweep by examining a reference count value in the buffer header. If the reference count is not 0, the buffer
stays in the pool and its reference count is adjusted downward in preparation for the next sweep; otherwise, the buffer
is made available for reuse: it is written to disk if dirty, removed from the hash lists, and put on a special list of buffers
called the free list.
NOTE
The set of buffers that the lazywriter sweeps through is sometimes called the LRU (for least recently used list).
However, it doesn't function as a traditional LRU because the buffers do not move within the list according to their
use or lack of use; the lazywriter clock hand does all the moving. Also note that the set of buffers that the lazywriter
inspects actually includes more than pages in the buffer pool. The buffers also include pages from compiled plans for
procedures, triggers, or ad hoc queries.

The reference count of a buffer is incremented each time the buffer's contents are accessed by any process. For data
or index pages, this is a simple increment by one. But objects that are expensive to create, such as stored procedure
plans, get a higher reference count that reflects their "replacement cost." When the lazywriter clock hand sweeps
through and checks which pages have been referenced, it does not use a simple decrement. It divides the reference
count by 4. This means that frequently referenced pages (those with a high reference count) and those with a high
replacement cost are "favored" and their count will not reach 0 any time soon, keeping them in the pool for further
use. The lazywriter hand sweeps through the buffer pool when the number of pages on the free list falls below its
minimum size. The minimum size is computed as a percentage of the overall buffer pool size but is always between
128 KB and 4 MB. SQL Server 2000 estimates the need for free pages based on the load on the system and the
number of stalls occurring. A stall occurs when a process needs a free page and none is available; the process goes to
sleep and waits for the lazywriter to free some pages. If a lot of stalls occur—say, more than three or four per
second—SQL Server increases the minimum size of the free list. If the load on the system is light and few stalls occur,
the minimum free list size is reduced and the excess pages can be used for hashing additional data and index pages or
query plans. The Performance Monitor has counters that let you examine not only the number of free pages but also
the number of stalls occurring.

User threads also perform the same function of searching for pages that can be placed on a free list. This happens
when a user process needs to read a page from disk into a buffer. Once the read has been initiated, the user thread
This document is created with the unregistered version of CHM2PDF Pilot

checks to see whether the free list is too small. (Note that this process consumes one page of the list for its own read.)
If so, the user thread performs the same function as the lazywriter: it advances the clock hand and searches for buffers
to free. Currently, it advances the clock hand through 16 buffers, regardless of how many it actually finds to free in
that group of 16. The reason for having user threads share in the work of the lazywriter is to distribute the cost across
all of the CPUs in an SMP environment. In fact, SQL Server 2000 actually has a separate free list for each CPU to
help further distribute the cost and improve scalability. A user thread that needs a free page first checks the free list for
the CPU it is running on; only if no pages are available will the user thread check the free lists for other CPUs.

Keeping pages in the cache permanently

You can specially mark tables so that their pages are never put on the free list and are therefore kept in memory
indefinitely. This process is called pinning a table. Any page (data, index, or text) belonging to a pinned table is never
marked as free and never reused unless it is unpinned. Pinning and unpinning is accomplished using the pintable
option of the sp_tableoption stored procedure. Setting this option to TRUE for a table doesn't cause the table to be
brought into cache, nor does it mark pages of the table as "favored" in any way; instead, it avoids the unnecessary
overhead and simply doesn't allow any pages belonging to that table to be put on the free list for possible
replacement.

Because mechanisms such as write-ahead logging and checkpointing are completely unaffected, such an operation in
no way impairs recovery. Still, pinning too many tables can result in few or even no pages being available when a new
buffer is needed. In general, you should pin tables only if you've carefully tuned your system, plenty of memory is
available, and you have a good feel for which tables constitute hot spots.

Pages that are "very hot" (accessed repeatedly) are never placed on the free list. A page in the buffer pool that has a
nonzero use count, such as a newly read or newly created page, is not added to the free list until its use count falls to
0. Before that point, the page is clearly hot and isn't a good candidate for reuse. Very hot pages might never get on
the free list, even if their objects aren't pinned—which is as it should be.

Checkpoints
Checkpoint operations minimize the amount of work that SQL Server must do when databases are recovered during
system startup. Checkpoints are run on a database-by-database basis. They flush dirty pages from the current
database out to disk so that those changes don't have to be redone during database recovery. A dirty page is one
that has been modified since it was brought from disk into the buffer pool. When a checkpoint occurs, SQL Server
writes a checkpoint record to the transaction log, which lists all the transactions that are active. This allows the
recovery process to build a table containing a list of all the potentially dirty pages.

Checkpoints are triggered when:

 A database owner explicitly issues a checkpoint command to perform a checkpoint in that database.
 The log is getting full (more than 70 percent of capacity) and the database is in SIMPLE recovery mode, or
the option trunc. log on chkpt. is set. (I'll tell you about recovery modes in Chapter 5.) A checkpoint is
triggered to truncate the transaction log and free up space. However, if no space can be freed up, perhaps
because of a long running transaction, no checkpoint occurs.
 A long recovery time is estimated. When recovery time is predicted to be longer than the recovery interval
configuration option, a checkpoint is triggered. SQL Server 2000 uses a simple metric to predict recovery
time because it can recover, or redo, in less time than it took the original operations to run. Thus, if
checkpoints are taken at least as often as the recovery interval frequency, recovery completes within the
interval. A recovery interval setting of 1 means that checkpoints occur at least every minute as long as
transactions are being processed in the database. A minimum amount of work must be done for the automatic
checkpoint to fire; this is currently 10 MB of log per minute. In this way, SQL Server doesn't waste time
taking checkpoints on idle databases. A default recovery interval of 0 means that SQL Server chooses an
appropriate value automatically; for the current version, this is one minute.
This document is created with the unregistered version of CHM2PDF Pilot

In a system with a large amount of memory, checkpoints can potentially generate lots of write operations to force all
the dirty pages out to disk. To limit the amount of resources checkpoints can consume, SQL Server limits checkpoint
operations to a maximum of 100 concurrent write operations. That might seem like a large number of writes, but on a
huge system, even 100 concurrent writes can take a long time to write out all the dirty pages. To optimize checkpoints
and to make sure that checkpoints don't need to do any more work than necessary, the checkpoint algorithm keeps
track of a generation number for each buffer in the cache. Without this number to help keep track of the work that's
been done, checkpoint operations could potentially write the same pages to disk multiple times.
Checkpoints and Performance Issues
A checkpoint is issued as part of an orderly shutdown, so a typical recovery upon restart takes only seconds. (An
orderly shutdown occurs when you explicitly shut down SQL Server, unless you do so via the SHUTDOWN WITH
NOWAIT command. An orderly shutdown also occurs when the SQL Server service is stopped through Service
Control Manager or the net stop command from an operating system prompt.) Although a checkpoint speeds up
recovery, it does slightly degrade run-time performance.

Unless your system is being pushed with high transactional activity, the run-time impact of a checkpoint probably
won't be noticeable. You can also use the sp_configure recovery interval option to influence checkpointing
frequency, balancing the time to recover vs. any impact on run-time performance. If you're interested in tracing how
often checkpoints actually occur, you can start your SQL Server with trace flag 3502, which writes information to
SQL Server's error log every time a checkpoint occurs.

The checkpoint process goes through the buffer pool, scanning the pages in buffer number order, and when it finds a
dirty page, it looks to see whether any physically contiguous (on the disk) pages are also dirty so that it can do a large
block write. But this means that it might, for example, write pages 14, 200, 260, and 1000 at the time that it sees
page 14 is dirty. (Those pages might have contiguous physical locations even though they're far apart in the buffer
pool. In this case, the noncontiguous pages in the buffer pool can be written as a single operation called a
gather-write. I'll define gather-writes in more detail later in this chapter.) As the process continues to scan the buffer
pool, it then gets to page 1000. Potentially, this page could be dirty again, and it might be written out a second time.
The larger the buffer pool, the greater the chance that a buffer that's already been written will get dirty again before the
checkpoint is done. To avoid this, each buffer has an associated bit called a generation number. At the beginning of
a checkpoint, all the bits are toggled to the same value, either all 0's or all 1's. As a checkpoint checks a page, it
toggles the generation bit to the opposite value. When the checkpoint comes across a page whose bit has already
been toggled, it doesn't write that page. Also, any new pages brought into cache during the checkpoint get the new
generation number, so they won't be written during that checkpoint cycle. Any pages already written because they're
in proximity to other pages (and are written together in a gather write) aren't written a second time.

Accessing Pages Using the Buffer Manager

The buffer manager handles the in-memory version of each physical disk page and provides all other modules access
to it (with appropriate safety measures). The memory image in the buffer pool, if one exists, takes precedence over
the disk image. If the page is dirty, the copy of the data page in memory includes updates that have not yet been
written to disk. When a page is needed for a process, it must exist in memory in the buffer pool. If the page isn't there,
a physical I/O is performed to get it. Obviously, because physical I/Os are expensive, the fewer physical I/Os you
have to perform the better. The more memory there is (the bigger the buffer pool), the more pages can reside there
and the more likely a page can be found there.

A database appears as a simple sequence of numbered pages. The dbid-fileno-pageno identifier uniquely specifies a
page for the entire SQL Server environment. When another module (such as the access methods manager, row
manager, index manager, or text manager) needs to access a page, it requests access from the buffer manager by
specifying the dbid-fileno-pageno identifier.

The buffer manager responds to the calling module with a pointer to the memory buffer holding that page. The
response might be immediate if the page is already in the cache, or it might take an instant for a disk I/O to complete
and bring the page into memory. Typically, the calling module also requests that the lock manager perform the
This document is created with the unregistered version of CHM2PDF Pilot

appropriate level of locking on the page. The calling module notifies the buffer manager if and when it is finished
dirtying, or making updates to, the page. The buffer manager is responsible for writing these updates to disk in a way
that coordinates with logging and transaction management.

Large Memory Issues

Systems with hundreds of megabytes of RAM are not uncommon. In fact, for benchmark activities, Microsoft runs
with a memory configuration of as much as 2 GB of physical RAM. Using the Enterprise Edition of SQL Server
allows the use of even more memory. The reason to run with more memory is, of course, to reduce the need for
physical I/O by increasing your cache-hit ratio.
Memory: How Much Is Too Much?
Some systems cannot benefit from huge amounts of memory. For example, if you have 2 GB of RAM and your
entire database is 1 GB, you won't even be able to fill the available memory, let alone benefit from its size. A pretty
small portion of most databases is "hot," so a memory size that's only a small percentage of the entire database size
can often yield a high cache-hit ratio. If you find that SQL Server is doing a lot of memory-intensive processing, such
as internal sorts and hashing, you can add additional memory. Adding memory beyond what's needed for a high
cache-hit ratio and internal sorts and hashes might bring only marginal improvement.

The Enterprise Edition of SQL Server 2000 running on Windows 2000 can use as much memory as Windows 2000
Advanced Server or Windows 2000 DataCenter Server allows by using the Windows 2000 Address Windowing
Extensions (AWE) API to support extra large address spaces. You must specifically configure an instance of SQL
Server to use the AWE extensions; the instance can then access up to 8 GB of physical memory on Advanced Server
and up to 64 GB on DataCenter Server. Although standard 32-bit addressing supports only 4 GB of physical
memory, the AWE extensions allow the additional memory to be acquired as nonpaged memory. The memory
manager can then dynamically map views of the nonpaged memory to the 32-bit address space.

You must be very careful when using this extension because nonpaged memory cannot be swapped out. Other
applications or other instances of SQL Server on the same machine might not be able to get the memory they need.
You should consider manually configuring the maximum amount of physical memory that SQL Server can use if you're
also going to enable it to use AWE. I'll talk more about configuring memory in Chapter 17.

Read ahead
SQL Server supports a mechanism called read ahead, whereby the need for data and index pages can be anticipated
and pages can be brought into the buffer pool before they're actually read. This performance optimization allows large
amounts of data to be processed effectively. Read ahead is managed completely internally, and no configuration
adjustments are necessary. In addition, read ahead doesn't use separate operating system threads. This ensures that
read ahead stays far enough—but not too far—ahead of the scan of the actual data.

There are two kinds of read ahead: one for table scans on heaps and one for index ranges. For table scans, the
table's allocation structures are consulted to read the table in disk order. Up to 32 extents (32 * 8 pages/extent *
8192 bytes/page = 2 MB) of read ahead are outstanding at a time. Four extents (32 pages) at a time are read with a
single 256-KB scatter read. (Scatter-gather I/O was introduced in Windows NT 4, Service Pack 2, with the Win32
functions ReadFileScatter and WriteFileGather.) If the table is spread across multiple files in a file group, SQL
Server has one read ahead thread per file. In SQL Server Standard Edition, each thread can still read up to 4 extents
at a time from a file, and up to 32 files can be processed concurrently. This means that the read ahead threads can
process up to 128 pages of data. In SQL Server Enterprise Edition, more extents can be read from each file, and
more files can be processed concurrently. In fact there is no set upper limit to number of extents or number of files;
SQL Server Enterprise Edition can read ahead enough data to fill 1 percent of the buffer pool.

For index ranges, the scan uses level one of the index structure (the level immediately above the leaf) to determine
which pages to read ahead. When the index scan starts, read ahead is invoked on the initial descent of the index to
minimize the number of reads performed. For instance, for a scan of WHERE state = 'WA', read ahead searches the
This document is created with the unregistered version of CHM2PDF Pilot

index for key = 'WA', and it can tell from the level one nodes how many pages have to be examined to satisfy the
scan. If the anticipated number of pages is small, all the pages are requested by the initial read ahead; if the pages are
contiguous, they're fetched in scatter reads. If the range contains a large number of pages, the initial read ahead is
performed and thereafter every time another 16 pages are consumed by the scan, the index is consulted to read in
another 16 pages. This has several interesting effects:
 Small ranges can be processed in a single read at the data page level whenever the index is contiguous.
 The scan range (for example, state = 'WA') can be used to prevent reading ahead of pages that won't be used
since this information is available in the index.
 Read ahead is not slowed by having to follow page linkages at the data page level. (Read ahead can be done
on both clustered indexes and nonclustered indexes.)
NOTE
Scatter-gather I/O and asynchronous I/O are available only to SQL Server running on Windows 2000 or Windows
NT. This includes the Personal Edition of SQL Server if it has been installed on Windows 2000 Professional or
Windows NT Workstation.

Merry-Go-Round Scans
The Enterprise Edition of SQL Server 2000 includes another optimization to improve the performance of nonordered
scans (a scan that isn't requested to be in any particular order), particularly if multiple nonordered scans of the same
table are requested simultaneously by different processes. Without this optimization, one process can start scanning
and get perhaps 20 percent of the way through the table before another process requests the same data. If the cache
is small, or is used by other processes for completely unrelated data, the pages scanned by the original process might
have been swapped out, which means that the buffer manager has to go back to disk to get the first pages of this table
again. When the original scanning process resumes, any pages that were read in ahead might be gone, and more disk
reads have to be done. This can cause some serious disk thrashing. A new optimization called Merry-Go-Round
scans allows SQL Server 2000 Enterprise Edition to avoid this thrashing by allowing the second process to start at
the same point the original process already reached. Both processes can then read the same data, and each page can
be read from disk only once and be used by both scans. When the first process finishes, the second process can read
the initial 20 percent of the table. Figure 3-5 illustrates a Merry-Go-Round scan.

Figure 3-5. SQL Server Enterprise Edition's Merry-Go-Round Scan.

The Log Manager

This document is created with the unregistered version of CHM2PDF Pilot

All changes are "written ahead" by the buffer manager to the transaction log. This means that the log records for a
transaction are always written to disk before the changed data pages are written. Write-ahead logging ensures that all
databases can be recovered to a consistent state, even in the event of a complete server failure, as long as the physical
medium (hard disk) survives. A process never receives acknowledgment that a transaction has been committed unless
it is on disk in the transaction log. For this reason, all writes to the transaction log are synchronous—SQL Server must
wait for acknowledgment of completion. Writes to data pages can be made asynchronously, without waiting for
acknowledgment, because if a failure occurs the transactions can be undone or redone from the information in the
transaction log.

The log manager formats transaction log records in memory before writing them to disk. To format these log records,
the log manager maintains regions of contiguous memory called log caches. In SQL Server 2000, log records do not
share the buffer pool with data and index pages. Log records are maintained only in the log caches.

To achieve maximum throughput, the log manager maintains two or more log caches. One is the current log cache, in
which new log records are added. In addition, there are one or more log caches available to be used when the current
log cache is filled. The log manager also has two queues of log caches: a flushQueue, which contains filled log caches
waiting to be flushed, and a freeQueue, which contains log caches that have no data (because they have been flushed)
and can be reused.

When a user process requires that a particular log cache be flushed (for example, when a transaction commits), the
log cache is placed into the flushQueue (if it isn't already there). Then the thread (or fiber) is put into the list of
connections waiting for the log cache to be flushed. The connection does not do further work until its log records have
been flushed.

The log writer is a dedicated thread that goes through the flushQueue in order and flushes the log caches out to disk.
The log caches are written one at a time. The log writer first checks to see whether the log cache is the current log
cache. If it is, the log writer pads the log cache to sector alignment and updates some header information. It then
issues an I/O event for that log cache. When the flush for a particular log cache is completed, any processes waiting
on that log cache are woken up and can resume work.
This document is created with the unregistered version of CHM2PDF Pilot

34

Transaction Logging and

Recovery
The transaction log records all changes made to the database and stores enough information to allow any changes to
be undone (rolled back) or redone (rolled forward) in the event of a system failure or if it is told to do so by the
application (in the case of a rollback command). Physically, the transaction log is a set of files associated with a
database at the time the database is created or altered. Modules that perform database updates write log entries that
exactly describe the changes made. Each log entry is labeled with a log sequence number (LSN) that is guaranteed to
be unique. All log entries that are part of the same transaction are linked together so that all parts of a transaction can
be easily located for both undo activities (as with a rollback) and redo activities (during system recovery).

The buffer manager guarantees that the transaction log is written before the changes to the database are written
(write-ahead logging). This is possible because SQL Server keeps track of its current position in the log by means of
the LSN. Every time a page is changed, the LSN corresponding to the log entry for that change is written into the
header of the data page. Dirty pages can be written to disk only when the LSN on the page is less than the LSN for
the last page written to the log. The buffer manager also guarantees that log pages are written in a specific order,
making it clear which log pages must be processed after a system failure, regardless of when the failure occurred. The
log records for a transaction are written to disk before the commit acknowledgement is sent to the client process, but
the actual changed data might not have been physically written out to the data pages. Therefore, although the writes to
the log must be synchronous (SQL Server must wait for them to complete so it knows that they're safely on disk),
writes to data pages can be asynchronous. That is, writes to data pages need only be posted to the operating system,
and SQL Server can check later to see that they were completed. They don't have to complete immediately because
the log contains all the information needed to redo the work, even in the event of a power failure or system crash
before the write completes. The system would be much slower if it had to wait for every I/O request to complete
before proceeding.

Logging involves demarcation of the beginning and end of each transaction (and savepoints, if a transaction uses
them). Between the beginning and ending demarcations, information exists about the changes made to the data. This
information can take the form of the actual "before and after" data values, or it can refer to the operation that was
performed so that those values can be derived. The end of a typical transaction is marked with a Commit record,
which indicates that the transaction must be reflected in the database or redone if necessary. A transaction aborted
during normal runtime (not system restart) due to an explicit rollback or something like a resource error (for example,
out of memory) actually undoes the operation by applying changes that undo the original data modifications. The
records of these changes are written to the log and marked as "compensation log records." If the system crashes after
a transaction commits but before the data is written out to the data pages, the transaction must be recovered. The
recovery process runs automatically at system startup. I'll continue to refer to recovery as a system startup function,
which is its most common role by far. However, recovery is also run during the final step of restoring a database from
backup and also can be forced manually.

There are some special issues regarding media recovery during a database restore that I won't discuss here. These
include the three recovery modes that you can set using the ALTER DATABASE statement and the ability to place a
named marker in the log to indicate a specific point to recover to. I'll talk about these issues in Chapter 5. The
discussion that follows deals with recovery in general, whether it's performed when the SQL Server Service is
restarted or when a database is being restored from a backup.

Recovery performs both redo (rollforward) and undo (rollback) operations. In a redo operation, the log is examined
and each change is verified as being already reflected in the database. (After a redo, every change made by the
transaction is guaranteed to have been applied.) If the change doesn't appear in the database, it is again performed
This document is created with the unregistered version of CHM2PDF Pilot

from the information in the log. Undo requires the removal of partial changes made by a transaction if the transaction
did not entirely complete.

During recovery, only changes that occurred or were still open (in progress) since the last checkpoint are redone or
undone. There are three phases to the recovery algorithm, and they are centered around the last checkpoint record in
the transaction log. The three phases are illustrated in Figure 3-6. The descriptions refer to a crash of SQL Server,
but these same recovery plans take place if SQL Server is intentionally stopped.
 Phase 1: Analysis The first phase is a forward pass starting at the last checkpoint record in the transaction
log. This pass determines and constructs a dirty page table (DPT) consisting of pages that might have been
dirty at the time of the crash (or when SQL Server stopped). An active transaction table is built that consists
of uncommitted transactions at the time of the crash.
 Phase 2: Redo This phase repeats history by returning the database to the state it was in at the time of the
crash. The starting point for this forward pass is the minimum of all the LSNs in the DPT. The DPT is used to
avoid reading pages that don't need recovering and to avoid overwriting nonlogged changes.
 Phase 3: Undo This phase moves backward from the end of the log, following the links between entries in
the transaction log for each transaction. Any transaction that was not committed at the time of the crash is
undone so that none of its changes are actually reflected in the database.

Figure 3-6. The three phases of the SQL Server recovery process.

Locking and Recovery

Locking, transaction management (rollback and rollforward), and recovery are all closely related. A transaction can
be rolled back only if all affected data was locked exclusively so that no other process could have seen changes in
progress (which might still be rolled back) or could have made changes to resources used by the transaction that
would prevent its being rolled back. Only one active transaction can modify a row at a time. This is why exclusive
locks must be held until a transaction is either committed or aborted. Until the moment it is committed, the transaction
logically does not exist. A transaction operating with Read Uncommitted isolation (dirty read) can sometimes read
data that logically never existed because it doesn't honor the existence of exclusive locks. But any other transaction
operating with a higher level of isolation (which occurs by default—operating with Read Uncommitted isolation must
be requested) would never allow such a phenomenon.

Page LSNs and Recovery

Every database page has an LSN in the page header that uniquely identifies it, by version, as rows on the page are
changed over time. This page LSN reflects the location in the transaction log of the last log entry that modified a row
on this page. During a redo operation of transactions, the LSN of each log record is compared to the page LSN of
the data page that the log entry modified; if the page LSN is less than the log LSN, the operation indicated in the log
entry is redone, as shown in Figure 3-7.

Because recovery finds the last checkpoint record in the log (plus transactions that were still active at the time of the
checkpoint) and proceeds from there, recovery time is short and all changes committed before the checkpoint can be
This document is created with the unregistered version of CHM2PDF Pilot

purged from the log or archived. Otherwise, recovery can take a long time and transaction logs become unreasonably
large. A transaction log cannot be purged beyond the point of the earliest transaction that is still open, no matter how
many checkpoints might have occurred subsequently. If a transaction remains open, the log must be preserved
because it's still not clear whether the transaction is done or ever will be done. The transaction might ultimately need to
be rolled back or rolled forward.

Some SQL Server administrators have noted that the transaction log seems unable to be purged to free up space,
even after the log has been archived. This problem often results from some process having opened a transaction,
which it then forgets about. For this reason, from an application development standpoint, you should ensure that
transactions are kept short. Another possible reason for this problem relates to a table being replicated using
transactional replication when the replication log reader hasn't processed it yet. This situation is less common, though,
because typically a latency of only a few seconds occurs while the log reader does its work. You can use DBCC
OPENTRAN to look for the earliest open transaction, or oldest replicated transaction not yet processed, and then
take corrective measures (such as killing the offending process or running the sp_repldone stored procedure to allow
the replicated transactions to be purged).

Figure 3-7. Comparing LSNs to decide whether to process the log entry.
This document is created with the unregistered version of CHM2PDF Pilot

34

The SQL Server Kernel and

Interaction with the Operating
System
The SQL Server kernel is responsible for interacting with the operating system. It's a bit of a simplification to suggest
that SQL Server has one module for all operating system calls, but for ease of understanding, you can think of it in this
way. All requests to operating system services are made via the Win32 API and C run-time libraries. When SQL
Server runs on Windows NT/2000, it runs entirely in the Win32 protected subsystem. Absolutely no calls are made in
Windows NT/2000 Privileged Mode; the calls are made in User Mode. This means that SQL Server cannot crash
the entire system, it cannot crash another process running in User Mode, and other such processes cannot crash SQL
Server. SQL Server has no device driver-level calls, nor does it use any undocumented calls to the operating system.
If the entire system crashes (giving you the so-called "Blue Screen of Death") and SQL Server happens to have been
running there, one thing is certain: SQL Server did not crash the system. Such a crash must be the result of faulty or
incompatible hardware, a buggy device driver operating in Privileged Mode, or a critical bug in the Windows
NT/2000 operating system code (which is doubtful).
NOTE
The Blue Screen of Death—a blue "bug check" screen with some diagnostic information—appears if a crash of
Windows NT/2000 occurs. It looks similar to the screen that appears when Windows NT initially boots up.
Exploiting the Windows NT/2000 Platform
Competitors have occasionally, and falsely, claimed that SQL Server must have special, secret hooks into the
operating system. Such claims are likely the result of SQL Server's astonishing level of performance. Yes, it is tightly
integrated with the operating systems on which it runs. But this integration is accomplished via completely public
interfaces—there are no secret "hooks" into the operating system. Yes, the product is optimized for Windows
NT/2000. That's SQL Server's primary platform. But other products could also achieve this level of optimization and
integration if they made it a chief design goal. Instead, they tend to abstract away the differences between operating
systems. Of course, there's nothing wrong with that. If Microsoft had to make sure that SQL Server could run on 44
different operating systems, it might also take a lowest-common-denominator approach to engineering. To do
anything else would be almost impossible. So although such an approach is quite rational, it is in direct conflict with the
goal of fully exploiting all services of a specific operating system. Since SQL Server runs exclusively on Microsoft
operating systems, it intentionally uses every service in the smartest way possible.

A key design goal of both SQL Server and Windows NT is scalability. The same binary executable files that run on
notebook computer systems run on symmetric multiprocessor super servers with loads of processors. Windows
NT/2000 is an ideal platform for a database server because it provides a fully protected, secure, 32-bit environment.
The foundations of a great database server platform are preemptive scheduling, virtual paged memory management,
symmetric multiprocessing, and asynchronous I/O. The operating system provides these, and SQL Server uses them
fully. The SQL Server engine runs as a single process on Windows NT/2000. Within that process are multiple threads
of execution. The operating system schedules each thread to the next processor available to run one.

Threading and Symmetric Multiprocessing

SQL Server approaches multiprocessor scalability differently than most other symmetric multiprocessing (SMP)
database systems. Two characteristics separate this approach from other implementations:
 Single-process architecture SQL Server maintains a single-process, multithreaded architecture that
This document is created with the unregistered version of CHM2PDF Pilot

reduces system overhead and memory use. This is called the Symmetric Server Architecture.
 Native thread-level multiprocessing SQL Server supports multiprocessing at the thread level rather than
at the process level, which allows for preemptive operation and dynamic load balancing across multiple
CPUs. Using multiple threads is significantly more efficient than using multiple processes.

To understand how SQL Server works, it's useful to compare its strategies to strategies generally used by other
products. On a nonthreaded operating system such as some UNIX variants, a typical SMP database server has
multiple DBMS processes, each bound to a specific CPU. Some implementations even have one process per user,
which results in a high memory cost. These processes communicate using Shared Memory, which maintains the cache,
locks, task queues, and user context information. The DBMS must include complex logic that takes on the role of an
operating system: it schedules user tasks, simulates threads, coordinates multiple processes, and so on. Because
processes are bound to specific CPUs, dynamic load balancing can be difficult or impossible. For the sake of
portability, products often take this approach even if they run on an operating system that offers native threading
services, such as Windows NT.

SQL Server, on the other hand, has a clean design with a single process and multiple operating system threads. The
threads are scheduled onto a CPU by a User Mode Scheduler, as discussed in Chapter 2. Figure 3-8 shows the
difference between SQL Server's threading architecture and that of other typical SMP database systems.

SQL Server always uses multiple threads, even on a single-processor system. Threads are created and destroyed
depending on system activity, so thread count is not constant. Typically, the number of active threads in SQL Server
ranges from 16 to 100, depending on system activity and configuration. A pool of threads handles each of the
Net-Libraries that SQL Server simultaneously supports, another thread handles database checkpoints, another
handles the lazywriter process, and another handles the log writer. A separate thread is also available for general
database cleanup tasks, such as periodically shrinking a database that is in autoshrink mode. Finally, a pool of threads
handles all user commands.

Figure 3-8. SQL Server's single-process, multiple-thread design.

The Worker Thread Pool

Although SQL Server might seem to offer each user a separate operating system thread, the system is actually a bit
more sophisticated than that. Because it's inefficient to use hundreds of separate operating system threads to support
This document is created with the unregistered version of CHM2PDF Pilot

hundreds of users, SQL Server establishes a pool of worker threads.

When a client issues a command, the SQL Server network handler places the command in a "completion queue" and
the next available thread from the worker thread pool takes the request. Technically, this queue is an operating system
facility called an IOCompletion port. The SQL Server worker thread waits in the completion queue for incoming
network requests to be posted to the IOCompletion port. If no idle worker thread is available to wait for the next
incoming request, SQL Server dynamically creates a new thread until the maximum configured worker thread limit has
been reached. The client's command must wait for a worker thread to be freed.

Even in a system with thousands of connected users, most are typically idle at any given time. As the workload
decreases, SQL Server gradually eliminates idle threads to improve resource and memory use.

The worker thread pool design is efficient for handling thousands of active connections without the need for a
transaction monitor. Most competing products, including those on the largest mainframe systems, need to use a
transaction monitor to achieve the level of active users that SQL Server can handle without such an extra component.
If you support a large number of connections, this is an important capability.
NOTE
In many cases, you should allow users to stay connected—even if they will be idle for periods of, say, an
hour—rather than have them continually connect and disconnect. Repeatedly incurring the overhead of the logon
process is more expensive than simply allowing the connection to remain live but idle.

A thread from the worker thread pool services each command to allow multiple processors to be fully utilized as long
as multiple user commands are outstanding. In addition, with SQL Server 2000, a single user command with no other
activity on the system can benefit from multiple processors if the query is complex. SQL Server can break complex
queries into component parts that can be executed in parallel on multiple CPUs. Note that this intraquery parallelism
occurs only if there are processors to spare—that is, if the number of processors is greater than the number of
connections. In addition, intraquery parallelism is not considered if the query is not expensive to run, and the threshold
for what constitutes "expensive" can be controlled with a configuration option called cost threshold for parallelism.
Active vs. Idle
SQL Server may consider many database connections idle, even though the associated human end user might be
quite active, filling in the data entry screen, getting information from customers, and so forth. But those activities don't
require any server interaction until a command is actually sent. So from the SQL Server engine perspective, the
connection is idle.

When you think of an active user vs. an idle user, be sure to consider the user in the context of the back-end database
server. In practically all types of applications that have many end users, at any given time the number of users who
have an active request with the database is relatively small. A system with 1000 active connections might reasonably
be configured with 150 or so worker threads. But this doesn't mean that all 150 worker threads are created at the
start—they're created only as needed, and 150 is only a high-water mark. In fact, fewer than 100 worker threads
might be active at a time, even if end users all think that they're actively using the system all the time.

Under the normal pooling scheme, a worker thread runs each user request to completion. If a given thread performs
an operation that causes a page fault, only that thread, and hence only that one client, is blocked. (A page fault occurs
if the thread makes a request for a memory page that isn't yet in RAM, so the virtual memory manager of the
operating system must swap that page in from disk. Such a request for memory must wait a long time relative to the
normal memory access time because a physical I/O is thousands of times more expensive than reading real memory.)

Now consider something more serious than a page fault. Suppose that while a user request is being carried out, a bug
is exposed in SQL Server that results in an illegal operation that causes an access violation (for example, the thread
tries to read some memory outside the SQL Server address space). The operating system immediately terminates the
offending thread—an important feature of a truly protected operating system. Because SQL Server makes use of
structured exception handling in Windows NT/2000, only the specific SQL Server user who made the request is
affected. All other users of SQL Server or other applications on the system are unaffected and the system at large will
This document is created with the unregistered version of CHM2PDF Pilot

not crash. Of course, such a bug should never occur and in reality is rare. But this is software, and software is never
perfect. Having this important reliability feature is like wearing a seat belt—you hope you never need it, but you're
glad it's there if a crash occurs.
NOTE
Since Windows 98 does not support SMP systems or thread pooling, the previous discussion is relevant only to
SQL Server running on Windows NT/2000. The following discussion of disk I/O is also relevant only to SQL Server
on Windows NT/2000.

Disk I/O in Windows NT/2000

SQL Server 2000 uses two Windows NT/2000 features to improve its disk I/O performance: scatter-gather I/O and
asynchronous I/O. The following descriptions were adapted from SQL Server Books Online:

Scatter-gather I/O As mentioned earlier, scatter-gather I/O was introduced in Windows NT 4, Service Pack 2.
Previously, all the data for a disk read or write on Windows NT had to be in a contiguous area of memory. If a read
transferred in 64 KB of data, the read request had to specify the address of a contiguous area of 64 KB of memory.
Scatter-gather I/O allows a read or write to transfer data into or out of discontiguous areas of memory.

If SQL Server 2000 reads in a 64-KB extent, it doesn't have to allocate a single 64-KB area and then copy the
individual pages to buffer cache pages. It can locate eight buffer pages and then do a single scatter-gather I/O that
specifies the address of the eight buffer pages. Windows NT/2000 places the eight pages directly into the buffer
pages, eliminating the need for SQL Server to do a separate memory copy.

Asynchronous I/O In an asynchronous I/O, after an application requests a read or write operation, Windows
NT/2000 immediately returns control to the application. The application can then perform additional work, and it can
later test to see whether the read or write has completed. By contrast, in a synchronous I/O, the operating system
doesn't return control to the application until the read or write completes. SQL Server supports multiple concurrent
asynchronous I/O operations against each file in a database. The maximum number of I/O operations for any file is
determined by the resources of the system, and SQL Server issues as many asynchronous, concurrent I/O requests as
possible.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
In this chapter, we've looked at the general workings of the SQL Server engine, including the key modules and
functional areas that make up the engine. I've also covered issues dealing with integration with Windows NT/2000. By
necessity, I've made some simplifications throughout the chapter, but the information should provide some insight into
the roles and responsibilities of the major subsystems in SQL Server, the general flow of the system, and the
interrelationships among subsystems.
This document is created with the unregistered version of CHM2PDF Pilot

34

Part III

Using Microsoft SQL Server

This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 4

Planning For and Installing SQL

Server
If you have appropriate hardware that's running Microsoft Windows, installing Microsoft SQL Server 2000 is a
snap. It requires little more than inserting the CD into the drive and answering a few questions. Even a novice can
comfortably install SQL Server in less than 15 minutes. The installation program doesn't ask a lot of questions about
how to configure your system because SQL Server 2000 is primarily self-configuring. I won't spend much time
discussing the mechanics of the installation program. You can consult the online Help and the documentation's
step-by-step guide if you need to. In fact, you can read the complete documentation on installation and upgrading
right from the CD before you take the first step in the installation process. The first splash screen that appears when
you load the CD into the drive contains an option to "Browse Setup/Upgrade Help." Installing SQL Server is so easy
that you might be tempted to dive in a bit too soon. That's OK if you're in "learning mode," but an installation that will
be part of a production system deserves some thoughtful planning.
This document is created with the unregistered version of CHM2PDF Pilot

34

SQL Server Editions

SQL Server is available in seven editions. Most are available on their own CD. The following descriptions are taken
from SQL Server Books Online:
 Enterprise Edition Used as a production database server. Supports all features available in SQL Server
2000 and scales to the performance levels required to support the largest Web sites and enterprise online
transaction processing (OLTP) and data warehousing systems.
 Standard Edition Used as a database server for a small workgroup or department.
 Personal Edition Used by mobile users who spend some of their time disconnected from the network but
run applications that require SQL Server data storage. Also used for stand-alone applications that require
local SQL Server data storage on a client computer.
 Developer Edition Used by programmers who develop applications that use SQL Server 2000 as their data
store. This edition supports all the features of the Enterprise Edition, but it is licensed for use only as a
development and test system, not a production server.
 Desktop Engine The redistributable version of the SQL Server 2000 database engine, which independent
software vendors can package with their applications.
 Windows CE Edition Used as the data store on Microsoft Windows CE devices. Capable of replicating
data with SQL Server 2000 Enterprise and Standard editions to keep Windows CE data synchronized with
the primary database.
 Enterprise Evaluation Edition A full-featured version available for free downloading from the Web.
Intended only for use in evaluating the features of SQL Server; stops running 120 days after installation.

Table 4-1, taken from SQL Server Books Online, summarizes the main differences between the editions and
indicates the database engine features that they support. You can read about other differences between the versions
(in areas such as replication, analysis services, and data transformation services) in SQL Server Books Online. Most
of this book will deal specifically with the Enterprise and Standard editions. Also, because the Developer Edition has
the same feature support as the Enterprise Edition (only the license agreement is different), most of the discussion will
also be relevant to the Developer Edition.

Any edition lets you install the full server and tools suite or just the tools. In fact, if you try to install an edition of SQL
Server on an OS for which it is not supported, all you will be able to install are the client tools. (For example, if you
try to install SQL Server Enterprise Edition on Windows 2000 Professional, you will get only the tools.)

Table 4-1. Features of the SQL Server editions.

Feature Personal Standard Enterprise
Runs on Microsoft
Windows NT 4 Server or Yes Yes Yes
Windows 2000 Server
Runs on Windows NT 4
Server, Enterprise Edition or
Yes Yes Yes
Windows 2000 Advanced
Server
AWE Support (Windows
No No Yes
2000 only)
SQL Server failover support No No Yes
Supports Microsoft Search
Yes, except on Windows
Service, full-text catalogs, Yes Yes
98
and full-text indexes
This document is created with the unregistered version of CHM2PDF Pilot

Maximum database size 2 GB 1,048,516 TB 1,048,516 TB

32 on Windows 2000
Datacenter Server 8 on
4 on all platforms except Windows NT 4 Server
Number of symmetric 2 on all platforms except
Windows NT 4 Server, Enterprise Edition and
multiprocessing (SMP) Windows 98, which
Enterprise Edition, which Windows 2000 Advanced
CPUs supports only 1
supports 8 Server 4 on Windows NT 4
Server and Windows 2000
Server
64 GB on Windows 2000
Datacenter Server 8 GB on
Windows 2000 Advanced
Server 4 GB on Windows
Physical memory supported 2 GB 2 GB
2000 Server 3 GB on
Windows NT 4 Server,
Enterprise Edition 2 GB on
Windows NT 4 Server

The Personal Edition has the same features as the Standard Edition, with only the following differences:
 The Personal Edition cannot support more than two processors; the Standard Edition supports four
processors on Windows 2000 Datacenter Server, Advanced Server and Server zand on Microsoft Windows
NT 4 Server. It supports up to eight processors on Windows NT 4 Server, Enterprise Edition.
 In the Personal Edition, the maximum size of a database is 2 GB. The maximum size in the Standard Edition is
1,048,516 TB.
 The Personal Edition can run on Microsoft Windows 98; the Standard Edition cannot.
 The Personal Edition is optimized for up to five concurrent users; the Standard Edition has practically no limit
on users.

Because of the limitations of the operating system, if the Personal Edition is installed on a machine running Windows
98, it also does not support:
 The Named Pipes and Banyan VINES server-side Net-Libraries. (The Windows 98 computer cannot
accept incoming connections using these protocols, but the client tools on such a computer can connect to
other SQL Servers on Windows NT and Windows 2000 using these protocols.)
 AppleTalk Net-Libraries.
 Windows Authentication for incoming server connections.
 The server side of Multiprotocol Net-Library encryption.
 Asynchronous I/O or scatter-gather I/O. (This includes true read-ahead scans. SQL Server issues what it
thinks are read-ahead requests, but these must be translated to synchronous calls because Windows 98
doesn't support asynchronous I/O. You get the benefit of the larger I/Os that read-ahead provides, but they
are synchronous.)
 Any component that corresponds to Windows NT and Windows 2000 Services. (The SQL Server database
engine and SQL Server Agent run as executable programs on Windows 98.)
 Event logs. (SQL Server uses a SQL Server Profiler-4-based mechanism to launch alerts on Windows 98.)
 SQL Server Performance Monitor.
 The SQL Server Version Upgrade utility.

I'll discuss most of these capabilities and tools in more detail later in the book.

Embedded SQL Server

If you want a version of SQL Server that you can embed in your own applications and redistribute, take a look at
Desktop Engine. Desktop Engine is the server component of SQL Server and includes basically the relational engine
and the storage engine. This edition of SQL Server 2000 is available on any of the CDs for the other editions in the
This document is created with the unregistered version of CHM2PDF Pilot

MSDE folder and also is available with Microsoft Office 10. Office 2000 users can download Desktop Engine, but
you need a Service Release version of Office 2000 to access Desktop Engine data from an Office 2000 product.

None of the usual SQL Server tools is installed with Desktop Engine; you're expected to use the visual tools that
come with Microsoft Visual Studio for creating objects. In addition, you can use all the usual tools that come with a
full SQL Server installation, such as SQL Server Enterprise Manager and SQL Query Analyzer, to access the data
stored in Desktop Engine.

Desktop Engine is also available as a set of Windows Installer merge modules, which independent software vendors
(ISVs) can use to install Desktop Engine during their own setup process. These modules can be merged into the ISV
setup program using available Windows Installer setup development tools. I won't discuss Desktop Engine further in
this book; see the documentation for additional details.
NOTE
When you install Microsoft Access 2000, you have the option of installing Desktop Engine or the JET database
engine for query processing. A whitepaper on the companion CD in the file Access Data Engine Options.doc
describes the differences between these two choices.
This document is created with the unregistered version of CHM2PDF Pilot

34

Hardware Guidelines
Because SQL Server runs on any hardware that runs Windows NT or Windows 2000, you can choose from among
thousands of hardware options. Although SQL Server Personal Edition runs on any hardware that runs Windows 98,
your production applications probably will be deployed on Windows NT or Windows 2000. Therefore, the following
discussion of hardware focuses on the needs of SQL Server running on Windows NT and Windows 2000 only.

Use Hardware on the Windows Hardware

Compatibility List
If you cobble together a system from spare parts, Windows NT and Windows 2000 will run just fine, but your
system might be unreliable. Unless you're a hobbyist and you like to tinker, this method isn't suitable for a production
system. Support is another issue. Many people buy motherboards, chassis, processors, hard drives, memory, video
cards, and assorted other peripherals separately and put together terrific systems, but the support options are limited.

Keep in mind that even name-brand systems occasionally fail. If you're using hardware that's on the Windows
Hardware Compatibility List (HCL) and you get a "blue screen" (Windows NT and Windows 2000 system crash),
you're more likely to be able to identify the problem as a hardware failure or find out, for example, that some known
device driver problem occurs with that platform (and that you can get an update for the driver). With a homegrown
system, such a problem can be nearly impossible to isolate. If you plan to use SQL Server, data integrity is probably a
key concern for you. Most cases of corrupt data in SQL Server can be traced to hardware or device driver failure.
(These device drivers are often supplied by the hardware vendor.)
NOTE
Over the life of a system, hardware costs are a relatively small portion of the overall cost. Using no-name or
homegrown hardware is penny-wise and pound-foolish. You can definitely get a good and cost-effective
HCL-approved system without cutting corners.

Performance = Fn (Processor Cycles, Memory, I/O

Throughput)
System throughput is only as fast as the slowest component: A bottleneck in one area reduces the rest of the system
to the speed of the slowest part. The performance of your hardware is a function of the processing power available,
the amount of physical memory (RAM) in the system, and the number of I/Os per second that the system can
support.

Of course, the most important aspect of performance is the application's design and implementation. You should
choose your hardware carefully, but there's no substitute for efficient applications. Although SQL Server can be a
brilliantly fast system, you can easily write an application that performs poorly—one that's impossible to "fix" simply
by "tuning" the server or by "killing it with hardware." You might double performance by upgrading your hardware or
fine-tuning your system, but application changes can often yield a hundredfold increase.

Unfortunately, there's no simple formula for coming up with an appropriately sized system. Many people ask for such
a formula, and minicomputer and mainframe vendors often provide "configuration programs" that claim to serve this
purpose. But the goal of those programs often seems to be to sell more hardware.

Once again, your application is the key. In reality, your hardware needs are dependent on your application. How
This document is created with the unregistered version of CHM2PDF Pilot

CPU-intensive is the application? How much data will you store? How random are the requests for data? Can you
expect to get numerous "cache hits" with the most frequently requested data often being found in memory, without
having to perform physical I/Os? How many users will simultaneously and actively use the system?

Invest in Benchmarking
If you're planning a large system, it's worthwhile to invest up front in some bench-marking. SQL Server has numerous
published benchmarks, most of them Transaction Processing Council (TPC) benchmarks. Although such benchmarks
are useful for comparing systems and hardware, they offer only broad guidelines and the benchmark workload is
unlikely to compare directly to yours. It's probably best to do some custom benchmarking for your own system.

Benchmarking can be a difficult, never-ending job, so keep the process in perspective. Remember that what counts is
how fast your application runs, not how fast your benchmark performs. And sometimes significant benchmarking
efforts are unnecessary—the system might perform clearly within parameters or similarly enough to other systems.
Experience is the best guide. But if you're testing a new application that will be widely deployed, the up-front cost of a
benchmark can pay big dividends in terms of a successful system rollout. Benchmarking is the developer's equivalent
of following the carpenter's adage "Measure twice, cut once."
SEE ALSO
For information on TPC benchmarks and a summary of results, see the TPC home page at http://www.tpc.org.

Nothing beats a real-world system test, but that's not always practical. You might need to make hardware decisions
before or while the application is being developed. In that case, you should develop a proxy test. Much of the work in
benchmarking comes from developing the appropriate test harness—the framework used to dispatch multiple clients
that can run the test programs simultaneously, to run the test programs for an allotted time, and to gather the results.
This document is created with the unregistered version of CHM2PDF Pilot

34

Hardware Components
The following general guidelines are no substitute for your own experience with your application, but they can help
you at the outset. For simplicity, the following sections assume that your server is basically dedicated to running SQL
Server, with little else going on. You can certainly combine SQL Server with other applications on the same machine,
and you can administer SQL Server locally using SQL Enterprise Manager. You can support multiple databases from
one SQL Server installation, and in SQL Server 2000, you can have multiple installations of SQL Server on the same
machine.

Table 4-2 (adapted from the SQL Server Books Online) shows the hardware requirements for installing SQL Server
2000.

Table 4-2. SQL Server 2000 hardware requirements.

Hardware Minimum requirements

Intel or compatible.
Computer
Pentium 166 MHz or higher.

Enterprise Edition: 64 MB minimum; 128 MB or more

recommended.

Standard Edition: 64 MB minimum.

Memory (RAM)1 Personal Edition: 64 MB minimum on Windows 2000;

32 MB minimum on all other operating systems.

Developer Edition: 64 MB minimum.

Desktop Engine: 64 MB minimum on Windows 2000; 32

MB minimum on all other operating systems.

SQL Server database components: 95 to 270 MB;

250 MB typical.

Hard disk space2 Analysis Services: 50 MB minimum; 130 MB typical.

English Query: 80 MB.

Desktop Engine only: 44 MB.

VGA or higher resolution.

Monitor
800 x 600 or higher resolution required for the SQL
Server graphical tools.
Pointing device Microsoft Mouse or compatible.
CD-ROM drive Required.
This document is created with the unregistered version of CHM2PDF Pilot

1 Additional memory might be required, depending on operating system requirements.

2 Actual requirements will vary based on your system configuration and the applications and features you choose to
install.

The Processor
SQL Server is CPU-intensive—more so than you might think if you assume that it is mostly I/O-constrained. Plenty
of processing capacity is a good investment. Your system uses a lot of processing cycles to manage many
connections, manage the data cache effectively, optimize and execute queries, check constraints, execute stored
procedures, lock tables and indexes, and enforce security. Because SQL Server typically performs few floating-point
operations, a processor that is fabulous with such operations but mediocre with integer operations is not the ideal
choice. Before you choose a processor, you should check benchmarks such as SPECint (integer)—not SPECfp
(floating-point) or any benchmark that's a combination of floating-point and integer performance. Performance on
SPECint correlates to the relative performance of SQL Server among different processor architectures, given
sufficient memory and I/O capacity. The correlation is a bit loose, of course, because other factors (such as the size of
the processor cache, the system bus efficiency, and the efficiency of the optimizing compiler used to build the system
when comparing processor architectures) also play an important role. But integer-processing performance is a key
metric for SQL Server.

SQL Server is built for SMP, but if your system is dedicated to SQL Server and supports fewer than 100
simultaneous users, a single CPU system will likely meet your needs. If you work with large data-warehousing
applications, you can use an SMP system effectively with a small number of users. SQL Server 2000 can run
complex queries in parallel, which means that the same query is processed simultaneously by multiple processors,
each doing a subset of the computations. SQL Server executes queries on multiple processors if sufficient processors
are available at the time of execution. (You'll see more details about how to choose the number of processors in
Chapter 15.) If you're uncertain about your needs or you expect them to grow, get a single-processor system that can
be upgraded to a dual (or even a quad) system.

For your server platform, you should buy the fastest single-processor system in its class. Buy tomorrow's technology,
not yesterday's. With many other applications, the extra processing power doesn't make much difference, but SQL
Server is CPU-intensive.
TIP
Remember that the processor's cache affects its performance. A big L2 (level 2, or secondary) cache is a good
investment. You might see a difference of 20 percent or more simply by using a 512 KB L2 cache instead of a 256
KB L2 cache in a given system. The cost difference is perhaps $100, so this is money well spent.

If your application is already available or you can conduct a fairly realistic system test, you can be much more exacting
in evaluating your hardware needs. Using the Windows Performance Monitor, simply profile the CPU usage of your
system during a specified period of time. (Use the Processor object and the % Processor Time counter.) Watch all
processors if you're using an SMP system. If your system is consistently above 70 percent usage or can be expected
to grow to that level soon, or if it frequently spikes to greater than 90 percent for 10 seconds or more, you should
consider getting a faster processor or an additional processor. You can use the SPECint95 guidelines to determine the
relative additional processing power that would be available by upgrading to a different or an additional processor.

Memory
Memory is used for many purposes in SQL Server 2000; I mentioned some of those purposes in Chapter 3, and I'll
discuss memory usage further in Chapter 17. By default, SQL Server 2000 automatically configures the amount of
memory it uses based on its needs and the needs of the operating system and other running applications.

If the SQL Server buffer manager can find data in the cache (memory), it can avoid physical I/Os. As discussed
This document is created with the unregistered version of CHM2PDF Pilot

earlier, retrieving data from memory is tens of thousands of times faster than the mechanical operation of performing a
physical I/O. (Physical I/O operations also consume thousands more CPU clock cycles than simply addressing real
memory.) As a rule, you should add physical memory to the system until you stop getting significant increases in cache
hits—or until you run out of money. You can easily monitor cache hits by using Performance Monitor. (For more
information, see Chapter 17.) Using the SQLServer:Buffer Manager object, watch the Buffer Cache Hit Ratio
counter. If adding physical memory makes the buffer cache-hit ratio increase significantly, the memory is a good
investment.

As mentioned earlier, by default SQL Server automatically adjusts its memory use to meet its needs and the needs of
the rest of the system. However, an administrator can override this and configure SQL Server to use a fixed amount
of memory. You should be extremely cautious if you decide to do this. Configuring memory appropriately is crucial to
good performance; configuring too little or too much memory can drastically impair performance. Too little memory
gives you a poor buffer cache-hit ratio. Too much memory means that the combined working sets of the operating
system and all applications (including SQL Server) will exceed the amount of real memory, requiring memory to be
paged to and from disk. You can monitor paging in Performance Monitor using statistics such as the Memory object's
counter for Pages/Sec.

Fortunately, adding physical memory to systems is usually quite simple (you pop in some SIMMs) and economical.
You should start conservatively and add memory based on empirical evidence as you test your application. SQL
Server needs no single correct amount of memory, so it's pointless to try to establish a uniform memory configuration
for all SQL Server applications that you might ever deploy.

SQL Server can run with a minimum 32 MB system (64 MB for SQL Server Enterprise Edition), but as is typically
the case with a minimum requirement, this is not ideal. If your budget isn't constrained, you should consider starting
with at least 128 MB of RAM. (If your budget is flexible or you believe that your application will be more demanding,
you might want to bump up memory to 256 MB. But it's probably best not to go higher than this initially without some
evidence that your SQL Server application will use it.)

When you buy your initial system, give some consideration to the configuration of the memory SIMMs, not only to
the total number of megabytes, so that you can later expand in the most effective way. If you're not careful, you can
get a memory configuration that makes it difficult to upgrade. For example, suppose your machine allows four
memory banks for SIMMs and you want 64 MB of memory that you might upgrade in the future. If your initial
configuration of 64 MB was done with four 16MB SIMMs, you're in trouble. If you want to upgrade, you have to
remove some or all of the 16-MB SIMMs and buy new memory. But if you configure your 64 MB as two 32-MB
SIMMs, you can add another one or two such SIMMs to get to 96 MB or 128 MB without replacing any of the
initial memory. Starting with one 64-MB SIMM gives you even more flexibility in upgrading.

When deciding on the initial amount of memory, you should also consider whether your system will be more
query-intensive or more transaction-intensive. Although more memory can help either way, it typically benefits
query-intensive systems most, especially if part of the data is "hot" and often queried. That data tends to remain
cached, and more memory allows more of the data to be accessed from the cache instead of necessitating physical
I/Os. If you have a query-intensive system, you might want to start with more memory than 128 MB. You can use the
memory not only to keep more of the needed data in cache, but also for some of the SQL Server 2000 query
processing techniques, such as hash joins, that work best with large amounts of memory. In contrast, a
transaction-intensive system might get relatively few cache hits, especially if the transactions are randomly dispersed or
don't rely much on preexisting data. In addition, the query processing techniques used for transaction processing are
typically much simpler and can't take advantage of extra memory for the actual processing. In such a system, the
amount of memory needed might be low and adding a faster I/O subsystem might make more sense than adding more
memory.

Disk Drives, Controllers, and Disk Arrays

Obviously, you need to acquire enough disk space to store all your data, plus more for working space and system
files. But you shouldn't simply buy storage capacity—you should buy I/O throughput and fault tolerance as well. A
This document is created with the unregistered version of CHM2PDF Pilot

single 8 GB drive stores as much as eight 1 GB drives, but its maximum I/O throughput is likely to be only about
one-eighth as much as the combined power of eight smaller drives (assuming that the drives have the same speed).
Even relatively small SQL Server installations should have at least two or three physical disk drives rather than a single
large drive. Microsoft has benchmarked systems that can do, say, 1000 transactions per second (tps) (in a
Debit-Credit scenario) using more than 40 physical disk drives and multiple controllers to achieve the I/O rates
necessary to sustain that level of throughput.

If you know your transaction level or can perform a system test to determine it, you might be able to produce a good
estimate of the number of I/Os per second you need to support during peak use. Then you should procure a
combination of drives and controllers that can deliver that number.

Typically, you should buy a system with SCSI drives, not IDE or EIDE drives. An IDE channel supports only two
devices. An EIDE setup consists of two IDE channels and can support four (2 × 2) devices. But more important, an
IDE channel can work with only a single device at a time—and even relatively small SQL Server installations usually
have at least two or three disk drives. If multiple disks are connected on an IDE channel, only one disk at a time can
do I/O. This is especially problematic if the channel also supports a slow device such as a CD-ROM drive. Since the
average access time for data on CDs is many times slower than for data on disk, your disk I/O would be delayed
while the CD is accessed.

SCSI is also a lot smarter than IDE or EIDE. A SCSI controller hands off commands to a SCSI drive and then
logically disconnects and issues commands to other drives. If multiple commands are outstanding, the SCSI controller
queues the commands inside the associated device (if the device supports command queuing) so that the device can
start on the next request as soon as the current request finishes. When a request finishes, the device notifies the
controller. A SCSI channel can support multiple fast drives and allow them to work at full speed. Slow devices such
as CD-ROM drives and tape drives do not hog the SCSI channel because the controller logically disconnects. A
single SCSI channel can support many drives, but from a practical standpoint, you shouldn't go beyond a 5:1 ratio if
you need to push the drives close to capacity. (That is, you should add an additional channel for every five drives or
so.) If you're not pushing the drives to capacity, you might be able to use as many as eight to ten drives per
channel—even more with some high-end controllers or random I/O patterns.

In TPC benchmarks (which are I/O-intensive, and in which the I/O is deliberately random), Microsoft generally
follows a 6:1 ratio of drives per SCSI channel. (Since most of the standard benchmarks measure both performance
and cost, Microsoft must balance the cost and benefits, just as they would for a real application.) Most real
applications don't have such perfectly random I/O characteristics, however. When the I/O is perfectly random, a
significant amount of the I/O time is spent with the disk drives seeking data rather than transferring data, which
consumes the SCSI channel. Seeking is taxing only on the drive itself, not on the SCSI channel, so the ratio of drives
per channel can be higher in the benchmark.

Note that this discussion is in terms of channels, not controllers. Most controllers have a single SCSI channel, so you
can think of the two terms as synonymous. Some high-end cards (typically RAID controllers, not just standard SCSI)
are dual-channel cards.
Some History
The original acronym RAID apparently stood for "redundant array of inexpensive disks," as evidenced in the paper
presented to the 1988 ACM SIGMOD by Patterson, Gibson, and Katz called "A Case for Redundant Arrays of
Inexpensive Disks (RAID)." However, the current RAIDbook, published by the RAID Advisory Board, uses
independent, which seems like a better description because the key function of RAID is to make separate, physically
independent drives function logically as one drive.

RAID Solutions
Simply put, RAID solutions are usually the best choice for SQL Server for these reasons:
 RAID makes multiple physical disk drives appear logically as one drive.
 Different levels of RAID provide different performance and redundancy features.
This document is created with the unregistered version of CHM2PDF Pilot

 RAID provides different ways of distributing physical data across multiple disk drives.
 RAID can be provided as a hardware solution (with a special RAID disk controller card), but it doesn't
necessarily imply special hardware. RAID can also be implemented via software.
 Windows NT and Windows 2000 Server provides RAID levels 0 (striping), 1 (mirroring), and 5 (striping
with parity) for implementation with SQL Server.

Some RAID levels provide increased I/O bandwidth (striping), and others provide fault-tolerant storage (mirroring or
parity). With a hardware RAID controller, you can employ a combination of striping and mirroring, often referred to
as RAID-10 (and sometimes as RAID-1+0 or RAID-1&0). RAID solutions vary in how much additional disk space
is required to protect the data and how long it takes to recover from a system outage. The type and level of RAID
you choose depends on your I/O throughput and fault tolerance needs.

RAID-0
RAID-0, or striping, offers pure performance but no fault tolerance. I/O is done in "stripes" and is distributed among
all drives in the array. Instead of the I/O capacity of one drive, you get the benefit of all the drives. A table's hot spots
are dissipated, and the data transfer rates go up cumulatively with the number of drives in the array. So although an
18-GB disk might do 100 random I/Os per second, an array of eight 2.1-GB disks striped with Windows NT Server
or Windows 2000 Server's RAID-0 might perform more than 400 I/Os per second.
Choosing an Appropriate Backup Strategy
Although you can use RAID for fault tolerance, it's no substitute for regular backups. RAID solutions can protect
against a pure disk-drive failure, such as a head crash, but that's far from the only case in which you would need a
backup. If the controller fails, garbage data might appear on both your primary and redundant disks. Even more
likely, an administrator could accidentally clobber a table, a disaster (such as a fire, flood, or earthquake) could occur
at the site, or a simple software failure (in a device driver, the operating system, or SQL Server) could threaten your
data. RAID doesn't protect your data against any of those problems, but backups do. I'll talk about SQL Server's
options for backing up your data in Chapter 5.
NOTE
You could argue that RAID-0 shouldn't be considered RAID at all because it provides no redundancy. That might be
why it's classified as level 0.

Windows NT and Windows 2000 provides RAID-0; a hardware RAID controller can also provide it. RAID-0
requires little processing overhead, so a hardware implementation of RAID-0 offers at best a marginal performance
advantage over the built-in Windows NT and Windows 2000 capability. One possible advantage is that RAID-0 in
hardware often lets you adjust the stripe size, while the size is fixed at 64 KB in Windows NT and Windows 2000
and cannot be changed.

RAID-1
RAID-1, or mirroring, is conceptually simple: a mirror copy exists for every disk drive. Writes are made to the
primary disk drive and to the mirrored copy. Because writes can be made concurrently, the elapsed time for a write is
usually not much greater than it would be for a single, unmirrored device. The write I/O performance to a given logical
drive is only as fast as that of a single physical drive.

Therefore, you can at best get I/O rates of perhaps 100 I/Os per second to a given logical RAID-1 drive, as
opposed to the much higher I/O rates to a logical drive that can be achieved with RAID-0. You can, of course, use
multiple RAID-1 drives. But if a single table is the hot spot in your database, even multiple RAID-1 drives won't be of
much help from a performance perspective. (SQL Server 2000 lets you create filegroups within a database for,
among other purposes, placing a table on a specific drive or drives. But no range partitioning capability is available for
data within a table, so this option doesn't really give you the control you need to eliminate hot spots.) If you need fault
tolerance, RAID-1 is normally the best choice for the transaction log. Because transaction log writes are synchronous
and sequential, unlike writes to the data pages, they're ideally suited to RAID-1.
Hardware-Based vs. Software-Based RAID
This document is created with the unregistered version of CHM2PDF Pilot

If price and performance are your only criteria, you might find that software-based RAID is perfectly adequate. If
you have a fixed budget, the money you save on hardware RAID controllers might be better spent elsewhere, such as
on a faster processor, more disks, more memory, or an uninterruptible power supply.

Purely in terms of performance, it might not be wise to spend extra money on low-end hardware RAID (if you have
sufficient CPU capacity in the server), but mid-level to high-end RAID offers many other benefits, including hot
swappability of drives and intelligent controllers that are well worth the extra money—especially for systems that need
to run 24 hours a day, 7 days a week.

You might also combine hardware-based and software-based RAID to great advantage. For example, if you have
three hardware-based RAID units, each with seven 9 GB disks, with RAID-5 you'd have 6 x 9 = 54 GB in each, and
Windows NT and Windows 2000 would see three 54 GB disks. You can then use software-based RAID on
Windows NT and Windows 2000 to define a RAID-0 striped volume to get a single 162 GB partition on which to
create SQL Server databases.

RAID-1 can significantly increase read performance because a read can be obtained from either the primary device
or the mirror device. (A single read isn't performed faster, but multiple reads will have a faster overall time because
they can be performed simultaneously.) If one of the drives fails, the other continues and SQL Server uses the
surviving drive. Until the failed drive is replaced, fault tolerance is not available unless multiple mirror copies were
used.
NOTE
The Windows NT and Windows 2000 RAID software doesn't allow you to keep multiple mirror copies; it allows
only one copy. But some RAID hardware controllers offer this feature. You can keep one or more mirror copies of a
drive to ensure that even if a drive fails, a mirror still exists. If a crucial system is difficult to access in a timely way to
replace a failed drive, this option might make sense.

Windows NT and Windows 2000 Server provides RAID-1, and some hardware RAID controllers also provide it.
RAID-1 requires little processing overhead, so a hardware implementation of RAID-1 offers at best a marginal
performance advantage over the built-in Windows NT and Windows 2000 capability (although the hardware solution
might provide the ability to mirror more than one copy of the drive).

RAID-5
RAID-5, or striping with parity, is a common choice for SQL Server use. RAID-5 not only logically combines
multiple disks to act like one disk, but it also records extra parity information on every drive in the array (thereby
requiring only one extra drive). If any drive fails, the others can reconstruct the data and continue without any data
loss or immediate down time. By doing an Exclusive OR (XOR) between the surviving drives and the parity
information, the bit patterns for a failed drive can be derived on the fly.

RAID-5 is less costly to implement than mirroring all the drives because only one additional drive is needed rather
than the double drives that mirroring requires. Although RAID-5 is commonly used, it's often not the best choice for
SQL Server because it imposes a significant I/O hit on write performance. RAID-5 turns one write into two reads
and two writes to keep the parity information updated. This doesn't mean that a single write takes four or five times as
long, because the operations are done in parallel. It does mean, however, that many more I/Os occur in the system,
so many more drives and controllers are required to reach the I/O levels that RAID0 can achieve. So although
RAID-5 certainly is less costly than mirroring all the drives, the performance overhead for writes is significant. Read
performance, on the other hand, is excellent—it's almost equivalent to that of RAID-0.

RAID-5 makes sense, then, for the data portion of a database that needs fault tolerance, that is heavily read, and that
does not demand high write performance. Typically, you shouldn't place the transaction log on a RAID-5 device
unless the system has a low rate of changes to the data. (As mentioned before, RAID-1 is a better choice for the
transaction log because of the sequential nature of I/O to the log.)
This document is created with the unregistered version of CHM2PDF Pilot

RAID-5 requires more overhead for writes than RAID-0 and RAID-1 do, so the incremental advantage provided by
a hardware RAID-5 solution over the Windows NT and Windows 2000 software solution can be somewhat higher
because more work must be offloaded to the hardware. However, such a difference would be noticeable only if the
system were nearly at I/O capacity. If this is the case in your system, you probably shouldn't use RAID-5 in the first
place because of the write performance penalty. You're probably better served by the combination of RAID-0 and
RAID-1, known as RAID-10.

RAID-10
RAID-10, or mirroring and striping, is the ultimate choice for performance and recoverability. This capability is really
not a separate type of RAID, but rather a combination of RAID-1 and RAID-0. It is sometimes also referred to as
RAID-1+0 or RAID-1&0. A set of disks is striped to provide the performance advantages of RAID-0, and the
stripe is mirrored to provide the fault-tolerance features and increased read performance of RAID-1. Performance for
both writes and reads is excellent.

The most serious drawback to RAID-10 is cost. Like RAID-1, it demands duplicate drives. In addition, the built-in
RAID software in the operating system doesn't provide this solution, so RAID-10 requires a hardware RAID
controller. Some hardware RAID controllers explicitly support RAID-10, but you can achieve this support using
virtually any RAID controller by combining its capabilities with those of Windows NT and Windows 2000. For
example, you can set up two stripe sets of the same size and number of disks using hardware RAID. Then you can
use Windows NT and Windows 2000 mirroring on the two stripe sets, which Windows NT and Windows 2000 sees
as two drives of equal size. If you need high read and write performance and fault tolerance and you cannot afford an
outage or decreased performance if a drive fails, RAID-10 is the best choice.

A separate RAID-10 array is usually not an appropriate choice for the transaction log. Because the write activity
tends to be sequential and is synchronous, the benefits of multiple disks are not realized. A RAID-1 mirroring of the
transaction log is preferable and cheaper. With internal Debit-Credit benchmarks, Microsoft has shown that the
transaction log on a simple RAID-1 device is sufficient to sustain thousands of transactions per second—which is
likely more than you need. Log records are packed, and a single log write can commit multiple transactions (known
as group commit). In the Debit-Credit benchmarks, 40 transactions can be packed into a log record. At a rate of
100 writes per second (writes/second), the log on a RAID-1 mirror would not become a bottleneck until around
4000 tps.

In fact, this threshold is probably even higher than 4000 tps. For pure sequential I/O, 100 writes/second is a
conservative estimate. Although a typical disk drive is capable of doing 100 random I/Os per second, the rate for
pure sequential I/O—largely eliminating the seek time of the I/Os—is probably better than 200 writes/second.

In a few cases, a physically separate RAID-10 array for the log might make sense—for example, if your system
requires high online transaction processing (OLTP) performance but you also use replication of transactions. Besides
performing sequential writes to the log, the system also does many simultaneous reads of the log for replication and
might benefit from having the multiple disks available. (In most cases, however, the log pages are in cache, so
RAID-10 probably won't provide a significant benefit. However, if replication is in a "catch-up" mode because the
distribution database isn't available or for some other reason, some significant benefit might exist.)

RAID-01
Most of the discussion in the SQL Server documentation that concerns combining RAID0 and RAID-1 assumes that
two stripes will be created first, and then one of the stripes will be defined as the mirror of the other. However, there
is another possibility, which is to create multiple mirrored sets of disks first and then create a stripe from the mirrors.
The first solution is referred to as mirrored stripes, and is the RAID 10 solution I discussed in the previous section.
The second solution is referred to as striped mirrors and can be called RAID-01.
NOTE
Both RAID-10 and RAID-01 refer to combining the capabilities of RAID-0 (striping) with RAID-1 (mirroring) so
This document is created with the unregistered version of CHM2PDF Pilot

that speed and fault tolerance are achieved, without the overhead of computing a parity value. However, the
terminology is not standard. As mentioned previously, some sources will refer to RAID-1&0, or RAID-1+0. Some
use the term RAID-10 to refer to striping a set of mirrored volumes, and use RAID-01 to refer to mirroring a striped
volume, which is the opposite of the terminology I've been using. In many cases it might not matter which is done first,
striping or mirroring, at least for the discussion at hand. I hope that in situations where it does matter, the authors of
whatever sources you read will define the terms they use.

Choosing the Best RAID Solution

Table 4-3 summarizes the characteristics of various RAID configurations commonly used with SQL Server.
Obviously, innumerable configurations are possible, but this chart will help you predict the characteristics for
additional configurations as well as typical ones. You should reject Option 1 for all but the smallest environments
because with this option the entire system runs with one disk drive. This is not efficient even for a low-end system,
unless you have little write performance and enough memory to get a high cache-hit ratio. Unless you're running SQL
Server on a laptop, there's little excuse for using only one drive.

Option 2, two physical drives and no use of RAID, represents an entry-level or development system. It offers
far-from-ideal performance and no fault tolerance. But it is significantly better than having only a single drive. Because
you should separate your data from your log before you even think about RAID, putting your data on one drive and
your log on the other is the best you can do.

If you don't need fault tolerance and limited recovery ability is sufficient, Option 3—using RAID-0 for the data and
placing the transaction log on a separate physical device (without mirroring)—is an excellent configuration. It offers
peak performance and low cost. One significant downside, however, is that if the drive with the log fails, you can
recover the data only to the point of your last transaction log or database backup. Because of this, Option 3 is not
appropriate for many environments. On the other hand, if you lose the drive with the data, you can back up the log to
the point of failure using the NO_TRUNCATE option and then restore that last log backup as the last step in
restoring the whole database.

Option 4 is the entry level for environments in which it is essential that no data be lost if a drive fails. This option uses
RAID-0 (striping, without parity) for the devices holding the data and uses RAID-1 (mirroring) for the transaction log,
so data can always be recovered from the log. However, any failed drive in the RAID-0 array renders SQL Server
unable to continue operation on that database until the drive is replaced. You can recover all the data from the
transaction logs, but you experience an outage while you replace the drive and load your transaction logs. If a drive
fails on either a transaction log or its mirrored copy, the system continues unaffected, using the surviving drive for the
log. However, until the failed log drive is replaced, fault tolerance is disabled and you're back to the equivalent of
Option 3. If that surviving drive also fails, you must revert to your last backup and you'll lose changes that occurred
since the last transaction log or database backup.
This document is created with the unregistered version of CHM2PDF Pilot

Table 4-3. Characteristics of RAID configurations used with SQL Server.

Option 4 offers excellent performance—almost as good as Option 3. This might come as a surprise to those
concerned with the mirroring of the log. RAID-1, of course, turns a single write into two. However, since the writes
are done in parallel, the elapsed time for the I/O is about the same as it would be for a single drive, assuming that both
the primary and mirrored drives are physically separate and are used only for log operations. If you need to protect all
data but recovery time is not a major concern and an outage while you recover is acceptable (that is, if loading
transaction logs or database backups is acceptable), Option 4 is ideal.

If you cannot tolerate an interruption of operations because of a failed drive, you should consider some form of
RAID-1 or RAID-5 for your database (as well as for the master database and tempdb, the internal workspace
area), as described in the following discussion of Options 5 through 8.

Option 5 uses basic RAID-1, a one-for-one mirroring of drives for the data as well as the log. I don't recommend
this option because RAID-1 doesn't provide the performance characteristics of RAID-0. Recall that RAID-0 enables
multiple drives to operate logically as one, with additive I/O performance capabilities. Although RAID1 can
theoretically work well if you handcraft your data layout, in practice this approach has the same drawbacks as using
filegroups instead of RAID-0. (See the following sidebar regarding filegroups.) This high-cost option is usually not a
high-performing option, at least if the database has significant hot spots. It's appropriate only for specialized purposes
in which I/O patterns can be carefully balanced.
Data Placement Using Filegroups
Are you better off simply putting all your data on one big RAID-0 striped set, or should you use filegroups to
carefully place your data on specific disk drives? Unless you're an expert at performance tuning and really understand
your data access patterns, I don't recommend using filegroups as an alternative to RAID. To use filegroups to
This document is created with the unregistered version of CHM2PDF Pilot

handcraft your physical layout, you must understand and identify the hot spots in the database. And although
filegroups do allow you to place tables on certain drives, doing so is tricky. It's currently not possible to partition
certain ranges of a table to different disks, so hot spots often occur no matter how cleverly you designed your data
distribution. If you have three or more physical disk drives, you should probably use some level of RAID, not
filegroups. I'll discuss filegroups in detail in Chapter 5.

Option 6, using RAID-5 for the data and RAID-1 for the log, is appropriate if the write activity for the data is
moderate at most and the update activity can be sustained. Remember that RAID-5 imposes a significant penalty on
writes because it must maintain parity information so that the data of a failed drive can be reconstructed on the fly.
Although the system does remain usable while operating with a failed drive, I/O performance is terrible: each read of
the failed disk must read each of the other disks in the array and reconstruct the value from the failed drives, and each
write becomes a similar read from each of the other disks, plus a write, so throughput drops precipitously when a
drive fails. With RAID-5, if you lose another drive in the system before you replace the first failed drive, you can't
recover because the parity information can't continue to be maintained after the first drive is lost. Until the failed drive
is replaced and regenerated, continued fault-tolerance capability is lost. RAID5 uses fewer bytes for storage overhead
than are used by mirroring all drives, but because disk drives are inexpensive and disks are rarely full, the
performance tradeoff is often simply not worth it. Finally, RAID-5 has more exposure to operational error than
mirroring. If the person replacing the failed drive in a RAID-5 set makes a mistake and pulls a good drive instead of
the failed drive, the failed drive might be unrecoverable.

If you need high read and write performance, cannot lose any data, and cannot suffer an unplanned system outage
when a drive fails, Option 7—RAID-10 or RAID-01 and a simple RAID-1 mirror for your log—is the way to go. As
mentioned earlier, RAID-10 offers the ultimate combination of performance and fault tolerance. Its obvious
disadvantage is cost. Option 7 requires not only a doubling of disk drives but also special hardware array controllers,
which command a premium price.

Options 7 and 8 differ in whether additional mirrored copies are kept. Option 7 presents the typical case using either
RAID-10 or RAID-01. Although both RAID-10 and RAID-01 exhibit the same behavior when all the disks are
functional, there is a significant difference when things start to break, both in terms of the fault tolerance provided and
in terms of performance.

Fault tolerance comparison of RAID-10 and RAID-01 I can best describe the difference in fault tolerance
between RAID-10 and RAID-01 with an example. Suppose we have 20 disk drives. For RAID-10, we could first
form two stripes of ten disks each and then mirror one stripe on the other. For RAID-01, we could first define ten
mirrored volumes and then create a stripe from those ten volumes. Either system can survive a failure of a single disk.
But what about a second disk failure? What are the chances that a second drive failure will bring down the whole
system?

Using RAID-10, if one disk in each stripe fails, the entire system will fail. You can compare this to having ten red
balls and ten black balls in a bucket, and asking what the chances are of pulling out two balls with different colors in
the first two pulls. After you pull out the first ball, statistically you have a 10/19 chance to pull out a ball with the
opposite color in the second pull. In other words, there's a 52.6 percent chance that a second failure will bring the
system down.

Using RAID-01 and a stripe of ten mirrors, the system would break down only if two disks from the same mirror
crash. In our bucket of balls analogy, we have ten pairs of balls, each pair with a different color. What are the chances
of pulling out two balls with the same color in our first two pulls? After pulling out the first ball, statistically, we have a
1/19 chance to pull out a ball with the same color in our second pull, or a 5.3 percent chance that a second failure will
bring the system down. So we can conclude that the latter configuration, using a stripe of mirrors, is much safer in
terms of fault tolerance—about ten times safer.

Performance comparison of RAID-10 and RAID-01 If one of the disks in a mirrored stripe (RAID 10) fails, the
whole failed stripe is unusable, leaving ten operational disks; therefore, we potentially lose 50 percent read
performance when performing asynchronous I/O. On the other hand, if one of the disks in a striped mirror (RAID 01)
fails, only one disk is unusable, leaving 19 operational disks; therefore, we potentially lose only 5 percent read
This document is created with the unregistered version of CHM2PDF Pilot

performance when performing asynchronous I/O.

Option 7, using the combined capabilities of striping and mirroring (RAID-10 or RAID-01), is probably a better
choice for many sites that currently use RAID-5. The performance and fault-tolerance capabilities are significantly
better, and with today's low-priced hardware, the cost isn't prohibitive.

If you need the utmost in fault tolerance of drive failures and your application is extremely mission critical, you can
consider more than a single mirrored copy of the data and log. Option 8 is overkill for most environments, but hard
drive prices are low enough that this option isn't necessarily prohibitively expensive. If you have a server in a remote
offsite location and it's difficult to get a technician to the site in a timely manner, building in expensive fault-tolerance
capabilities might make sense.

The bottom line: buy the right system

By now, you should know there's no one correct answer to the RAID-level question. The appropriate decision
depends on your performance characteristics and fault tolerance needs. A complete discussion of RAID and storage
systems is beyond the scope of this book. But the most important point is that you shouldn't simply buy storage
space—you should buy the system that gives you the appropriate level of redundancy and fault tolerance for your
data.
SEE ALSO
For a detailed, authoritative discussion of RAID, consult the RAIDbook produced by the RAID Advisory Board.
For more information about the RAID Advisory Board, see its home page at http://www.raid-advisory.com.

Hardware or software RAID?

After choosing a RAID level, you must decide whether to use the RAID capabilities of your operating system or
those of a hardware controller. If you decide to use RAID10, the choice is simple. Windows NT Server RAID does
not currently allow you to keep more than one mirror copy of a drive, so you must use hardware array controllers.
Windows 98 doesn't support software-based RAID at all, and Windows NT Workstation and Windows 2000
Professional provide only RAID-0 support.

As for solutions using RAID-0 and RAID-1, the performance advantage of a hardware array controller over the
capabilities of the operating system might be minimal. If price and performance are your main criteria, you can use
standard fast-wide SCSI controllers and use the money you save to buy more drives or more memory. Several of the
formal TPC benchmarks submitted using SQL Server used only the Windows NT versions of RAID-0 and RAID-1
solutions (Option 4) and standard SCSI controllers and drives, with no special hardware RAID support. Hardware
RAID solutions provide the largest relative performance difference over the equivalent Windows NT and Windows
2000 software solution if you use RAID-5, but even in this case, the difference is usually marginal at best. Each
hardware RAID controller can cost thousands of dollars, and a big system still needs multiple controllers. Using
hardware controllers makes the most sense when you need the additional capabilities they provide, such as hot
swappability (discussed in the following section).

More About Drives and Controllers

No matter what RAID level you use, you should replace a failed drive as soon as possible. For this reason, you
should consider a system that offers hot swappable drives. These drives let you quickly replace the faulty drive
without shutting down the system. If the drive is part of a RAID-1 or RAID-5 set, SQL Server can continue to run
without error or interruption when the drive fails and even while it is being physically replaced. If you use the
Windows NT or Windows 2000 version of RAID1 or RAID-5, you must shut down SQL Server only briefly to
reestablish the fault tolerance for the drive because that process requires an exclusive lock on the disk drive by the
operating system. (For RAID-1 you'll need to break and re-establish the mirror, and for RAID-5 you'll need to
regenerate the striped set with parity.) When you restart SQL Server, the server computer won't even need to be
This document is created with the unregistered version of CHM2PDF Pilot

rebooted. (Some hardware RAID solutions might also be able to regenerate or remirror the drive "below" the level of
the operating system and hence not require SQL Server to be shut down.) Some systems also offer hot-standby
drives, which are simply extra drives that are already installed, waiting to take over. Either of these approaches can
get you back in business quickly.

Your disk controller must guarantee that any write operation reported to the operating system as successful will
actually be completed. You should never use a write-back caching controller that can "lie" (report a write as
completed without guaranteeing that it will actually perform the write) because your databases can become corrupted.
In a write-back cache scheme, performance is increased because the bits are simply written to the cache (memory)
and the I/O completion is immediately acknowledged. The controller writes the bits to the actual media a moment
later. This introduces a timing window that makes the system vulnerable unless the controller designer has protected
the cache with a battery backup and has provided correct recovery logic.

Let's say, for example, that a power failure occurs immediately after the controller reports that a write operation has
completed but before the cache is actually written to disk. With write-ahead logging, SQL Server assumes that any
change is physically written to its transaction log before it acknowledges the commit to the client. If the controller has
just cached the write and then fails and never completes it, the system's integrity is broken. If a controller provides
write-back caching as a feature, it must also guarantee that the writes will be completed and that they will be properly
sequenced if they are reported to the operating system as successful. To be reliable, the caching controller also must
guarantee that once it reports that an I/O has completed, the I/O will actually be carried out, no matter what. Such a
controller typically employs a built-in battery backup and has solutions that guarantee that any write reported as
completed will in fact be completed. If the system fails before a write operation has been completed, the operation is
maintained in the controller's memory and is performed immediately after the system restarts, before any other I/O
operations occur.

If a controller cannot guarantee that the write will ultimately be completed, you should disable the write-back caching
feature of the controller or use a different controller. Disabling write-back caching in an I/O-intensive environment
(more than 250 I/Os per second) might result in a SQL Server performance penalty of less than 5 percent. (After all,
those writes must be performed eventually. The caching reduces the latency, but ultimately just as much I/O must be
carried out.) In a less I/O-intensive test, the effect would likely be negligible. Realistically, the amount of the penalty
doesn't matter much if the write isn't guaranteed to complete. Write-back caching must be disabled, or you run the
risk of corrupting your data.

Uninterruptible Power Supply

If you're not using a write-back caching controller, you don't need UPS hardware to protect the integrity of your data
from a power outage. SQL Server recovers your system to a consistent state after any sudden loss of power.
However, you should definitely consider using a UPS; support for UPS hardware is built into Windows NT and
Windows 2000 Server. Most hardware failures, such as memory failures and hardware crashes, result from power
spikes that a UPS would prevent. Adding a UPS is probably the single most important thing you can do from a
hardware perspective to maximize the availability and reliability of your system.

Without a UPS, the machine reboots and perhaps does a CHKDSK of the file system after a power flicker or power
spike. This can take several minutes. During this time, your SQL Server is unavailable. A UPS can prevent these
interruptions. Even if a power outage lasts longer than the life of your UPS, you have time to do an orderly shutdown
of SQL Server and checkpoint all the databases. The subsequent restart will be much faster because no transactions
will need to be rolled back or rolled forward.
Battery Backup, UPS, and Caching
A UPS is not the same as the battery backup of the on-board controller RAM. The job of a UPS is to bridge power
failures and to give you time to do an orderly shutdown of the system. The computer continues to run for a while from
the power supplied by the UPS, and it can be shut down in an orderly way (which the UPS software will do) if power
isn't restored within the expected battery life.
This document is created with the unregistered version of CHM2PDF Pilot

The battery backup on a caching controller card ensures that the contents of the controller's volatile memory survive;
the memory is not lost, so when the system restarts, the write completes. But it doesn't provide the power to actually
ensure that a write succeeds. Some cards even allow the battery backed-up RAM to be moved to a different
controller if the initial controller fails. Some provide two sets of batteries so that one can be replaced while the other
maintains the RAM's contents, theoretically allowing you to keep adding new batteries indefinitely.

You might expect that using a caching controller that doesn't have a battery backup in combination with a UPS is
generally OK if you do an orderly shutdown of SQL Server during the life of the UPS's power. However, this isn't
always the case. SQL Server support engineers who get customers' servers back up and running can tell horror
stories about customers who use write-caching controllers that were not designed around transactional integrity. The
caching controller must take into account issues besides power. For example, how does the controller respond to the
user pressing the Reset button on the computer? A couple of years ago, a well-known caching controller would have
dumped its cache if you hit the Reset button—those writes were never carried out even though the calling application
(SQL Server) was notified that they had been carried out. This situation often led to corrupt databases.

The Disk Subsystem

The reliability of the disk subsystem (including device drivers) is vital to the integrity of your SQL Server data. To
help you uncover faulty disk systems, the SQL Server support team created a stand-alone, I/O-intensive,
multithreaded file system stress test application. This application exercises the Win32 overlapped (asynchronous) I/O
services, opens its files with the same write-through cache flags used by SQL Server, and performs similar types of
I/O patterns. But the test is totally distinct from SQL Server. If this non-SQL Server I/O test cannot run without I/O
errors, you do not have a reliable platform on which to run SQL Server, period. Of course, like any diagnostic
program, the test might miss some subtle errors. But your confidence should certainly be higher than it would be
without testing. You can download the test program for free from the Microsoft Support site at
http://support.microsoft.com. On the Knowledge Base Search page, select All Microsoft Products in Step 1. In Step
2, select Specific Article ID Number. In Step 3, enter Q231619. Click Go. Open the Knowledge Base article that's
returned ("INF: SQL70IOStress Utility to Stress Disk Subsystem" and download the file called SQL70IOStress.exe.
SQL70IOStress.exe is also included on the companion CD.

Fallback Server Capability

SQL Server Enterprise Edition can take advantage of Microsoft Cluster Server (MSCS), which is available on
Windows NT 4 Server, Enterprise Edition, Windows 2000 Advanced Server, and Windows 2000 Datacenter
Server. You can install SQL Server 2000 on multiple nodes in a cluster, and all nodes are seen as having a single
name on the network. Applications do not have to be aware that SQL Server is installed on a cluster, and MSCS
handles the failover completely invisibly to users of SQL Server. For more details, see the operating system
documentation.

MSCS requires specialized hardware. By using various RAID options, you can protect your data without clustering.
But without clustering, you are not protected against an application outage if the machine fails. MSCS can
automatically shift the control of a "switchable" hard-drive array from a damaged node in the cluster to another node.
SQL Server 2000 supports two-node clustering on Windows NT Server 4, Enterprise Edition and Windows 2000
Advanced Server, and up to four-node clustering on Windows 2000 Datacenter Server. Only one of the connections
to the hard-drive array is active at any time. One of the other hard-drive connections becomes active only if the
system detects that the computer currently in control of the hard drive has shut down because of a hardware failure.

The SQL Server installation on the other nodes in the cluster can perform useful work while the first installation of
SQL Server operates normally and accesses the data on the shared disk. The other servers can run SQL Server
(using different databases than those used by the primary server), they can act as file and print servers, or they can run
some other application. If another server needs to take over the work of the first server, that second server must have
enough resources to add the primary server's workload to its existing workload.
NOTE
This document is created with the unregistered version of CHM2PDF Pilot

Keep in mind that even with clustering, you need RAID to protect against media failure. The log, at the very least,
must be mirrored to protect against a disk failure. The second server can run the disk drives of the failed first server,
but those drives must also be protected against media failure.

Most applications probably do not need clustering because today's hardware is highly reliable. The most common
hardware failure occurs in a disk drive, which can be protected without clustering by using RAID. As with RAID,
clustering does not reduce the need for comprehensive and rigorous backup procedures. You'll probably need
backups to recover from an occasional administrator mistake (human error), a catastrophe such as a fire or flood at
the site, or a software anomaly. But some applications are so critical that the redundancy of clustering is necessary.

Other Hardware Considerations

If the server is mostly a "lights-out" operation and you won't be administering it locally or running other applications
from the actual server machine, it doesn't make much sense to buy a first-class monitor or video card. Consider using
an electronic switch box and sharing a monitor, keyboard, and mouse among multiple servers. If you don't use
multiple servers, use a basic VGA monitor and video card.

From a performance perspective, a single high-quality network card is sufficient to handle virtually every SQL Server
installation. Windows NT and Windows 2000 Server can support multiple network cards simultaneously, and
sometimes your network topology will compel you to use this configuration with SQL Server as well. (Perhaps you're
supporting both Ethernet and token ring clients on the same server.) However, one network card usually is more than
enough—some sites support hundreds or thousands of users with a single 10-Mbit Ethernet card.

When you plan your hardware configuration, be sure to plan for your backup needs. You can back up to either tape
or disk devices. As with regular I/O, you'll frequently need multiple backup devices to sustain the I/O rates required
for backup. SQL Server can stripe its backup to multiple devices, including tape devices, for higher performance. For
the same reasons mentioned earlier when I discussed disk devices, a SCSI tape device is typically preferable to an
IDE/EIDE device.
NOTE
Not long ago, backup meant tape. Today, with disk space being the least expensive component of the entire system,
many sites configure their hardware with additional disk devices purely for backup use because they prefer the extra
speed and ease that backup to disk offers over tape backup. Some sites even use hot swappable drives for their
backup; they rotate between a few sets of such drives, keeping a set off site in case of disaster.
This document is created with the unregistered version of CHM2PDF Pilot

34

The Operating System

The SQL Server engine runs best on Windows NT and Windows 2000 Server, and your production applications are
meant to run on SQL Server on Windows NT 4 Server, Enterprise Edition, Windows NT and Windows 2000
Server, Windows 2000 Advanced Server, or Windows 2000 Datacenter Server. Although you can install the
Personal Edition of SQL Server on Windows NT 4 Workstation, Windows 2000 Professional, or Windows 98,
many SQL Server capabilities are not supported on the Personal Edition. (I discussed these limitations earlier in the
chapter.) Memory management and the process scheduling behavior on Windows NT and Windows 2000 Server are
optimized for running services such as SQL Server. Windows NT and Windows 2000 Workstation and Windows
98, on the other hand, are more suited to running desktop applications. The simple fact is that SQL Server runs better
with medium to large workloads on Windows NT and Windows 2000 Server than on any other platform.

Although you might get SQL Server to work on a platform other than its supported ones, you should stick with the
supported platforms for the following reasons:
 Only supported platforms get significant test coverage. An unsupported platform might seem to work, but a
subtle problem might occur later.
 Authorized SQL Server support providers and Microsoft SQL Server Product Support Services (PSS) focus
on the supported platforms. If your problem is specific to an unsupported platform, you might not be able to
get enough help.
 Typically, a new release of SQL Server is targeted at the most recent (or an impending) release of Microsoft's
Server operating system and also supports the prior major release. As of this writing, Windows 2000 has
been released, so SQL Server 2000 runs on Windows NT 4 (with Service Pack 5) and Windows 2000. It is
not supported on Windows NT Server 3.51 or earlier. If you try to run Setup on a Windows NT version
earlier than version 4 with Service Pack 5, you'll get an error and will be prevented from continuing.

You can install SQL Server on any Windows NT and Windows 2000 Server in your domain, but for the best
performance, do not install it on any of your domain controllers. Although SQL Server will run fine on a domain
controller, the controller's tasks of maintaining and replicating the Windows NT and Windows 2000 accounts
database take up processing cycles and memory. In less resource-intensive environments, this usually doesn't make
much difference, but if your system is part of a large domain, it's better to separate SQL Server processing from those
activities.
This document is created with the unregistered version of CHM2PDF Pilot

34

The File System

The performance difference between running SQL Server on the two major Windows NT and Windows 2000 file
systems—NTFS and FAT—is minimal. This isn't surprising because SQL Server manages the structures within its
database files without much use of the operating system.

I recommend NTFS as the file system of choice for SQL Server. It's more robust than FAT, and it's better able to
recover fully and quickly from an ungraceful system shutdown. The integrity of the file system is vital to the integrity of
your data, so this fact alone should drive most SQL Server users to NTFS. Also, the security features of NTFS are
vital if security is important to you. Unlike FAT, NTFS allows security and auditing on all files; these are important
ways to protect your data when SQL Server is not running. (SQL Server provides its own robust security.) When
SQL Server is not running, the operating system is your prime protection. If you have a legitimate need to dual-boot
your system to MS-DOS or to Windows 98, or if you need to use some disk utilities that work only with FAT, you
might choose FAT. (You can, of course, make the partition the system boots from a FAT partition and format the
other partitions with NTFS.)

Although it is physically possible to create SQL Server databases on NTFS compressed volumes, Microsoft does
not support this, so I can't recommend that you ever consider it. Other reasons for avoiding compressed drives, in
addition to Microsoft's lack of support, include performance degradation (over 50 percent in some conditions) and
the inability to provide reliable transaction recovery of your databases. Transactional recovery requires sector-aligned
write, and compressed volumes do not support this feature.

With previous versions of SQL Server, you could run SQL Server on a new partition that was not formatted with
any file system. Such a partition is commonly referred to as a raw partition. Although you cannot run SQL Server
2000 from a raw partition, you still have the option of creating database files on raw partitions. Storing data on raw
partitions is common for UNIX database products, and users who have UNIX DBMS backgrounds sometimes do
this with SQL Server as well. Under heavy workloads, raw partitions can provide a small (about two percent)
performance benefit. If the I/O system is not at capacity, you'll probably notice no performance difference at all.

For most installations, using raw partitions is not appropriate. Even if you gain some performance benefits by forgoing
a file system, you also forgo benefits that a file system provides, such as basic file operations and utilities (copy,
delete, rename, and dir) and important operations such as detecting and resolving bad disk sectors. If there's no file
system to perform bad sector operations, you must be sure that your hardware will take care of this. (This type of
hardware is available but costly.)
This document is created with the unregistered version of CHM2PDF Pilot

34

Security and the User Context

To install SQL Server on a machine, you need not be an administrator of the domain, but you must have
administrator privileges on the machine. Users can install most of the SQL Server client utilities without administrator
privileges.

Before you set up your system, give some thought to the user context in which SQL Server and SQL Server Agent
will run. A new SQL Server environment can set up the SQL Server engine to run in the context of the special system
(LocalSystem) account if it is not being installed on a Windows 2000 domain controller. This account is typically used
to run services, but it has no privileges on other machines, nor can it be granted such privileges. However, the default
is to have the SQL Server service and the SQL Server Agent service run in the context of a domain administrator
account. This allows SQL Server to more easily perform tasks that require an external security context, such as
backing up to another machine or using replication. If you don't have access to a domain administrator account, you
might want to set up SQL Server to run in the context of a local administrator. This will still give SQL Server sufficient
privileges to run on the local machine.

If you're not going to use the LocalSystem account, it's a good idea to change the account under which SQL Server
runs to a user account that you've created just for this purpose rather than use an actual local or domain administrator
account. The account must be in the local Administrators group if you're installing SQL Server on Windows NT or
Windows 2000. You can create this account before you begin installing SQL Server, or you can change the account
under which SQL Server runs at a later time. Changing the account is easy: you use the Services applet in the
Windows NT Control Panel. In Windows 2000, you can use the Microsoft Management Console, which is
accessible from Start/Programs/Administrative Tools/Computer Management. When you choose or create a user
account for running SQL Server, make sure you configure the account so that the password never expires; if the
password expires, SQL Server won't start until the information for the service is updated. (This is why I don't
recommend using the real Administrator account for SQL Server; that account will probably have its password
changed regularly.)

If you plan to use the mail integration features (SQL Mail), you should be aware of one additional issue: if you're
using Microsoft Exchange on the same machine as SQL Server, you should run SQL Server in the context of the user
account for which your Exchange client is configured. SQL Server can then pick up the Exchange configuration for
that user automatically.

By default, the installation program chooses to use the same account for both the SQL Server and the SQL Server
Agent services. SQL Server Agent needs a domain-level security context to connect to other computers in more
situations than does SQL Server. For example, if you plan to publish data for replication, SQL Server Agent needs a
security context to connect to the subscribing machines. If you won't be publishing data for replication or scheduling
tasks on SQL Server that require access to other computers, you can have SQL Server Agent run in the
LocalSystem account. If you specify a domain account but the domain controller cannot validate the account (because
the domain controller is temporarily unavailable, for example), go ahead and install using the LocalSystem account and
change it later using the Services applet (in Windows NT) or the Microsoft Management Console (in Windows
2000).

The accounts under which the SQL Server service and the SQL Server Agent service run can also be changed from
SQL Enterprise Manager. For the SQL Server service, right-click on the name of your SQL Server, choose
Properties, and then go to the Security tab. For the SQL Server Agent service, open the Management folder,
right-click on SQL Server Agent, choose Properties, and stay on the General tab.
This document is created with the unregistered version of CHM2PDF Pilot

34

Licensing
When you install SQL Server, you'll see a dialog box asking which licensing option you want to use: per-processor
or per-seat.
NOTE
The license agreement in the SQL Server box is a legal document that you should read and understand. This section
explains the licensing schemes but does not represent the official licensing policy of the product. Packaging and
licensing periodically change. Consult the licensing agreement to be sure you're in compliance and have chosen the
best options for your needs.

SQL Server Processor License

A Processor License includes access for an unlimited number of users to connect from inside the corporate LAN or
WAN or outside the firewall. Under the per-processor licensing model, a Processor License is required for each
processor running the SQL Server 2000 software. Processor Licenses can be used in any Internet, extranet, or
intranet scenario. No additional Server Licenses, Client Access Licenses, or Internet Connector Licenses will need to
be purchased.

Server Licenses and CALs

A Server License (for either Standard or Enterprise Edition) is required for every server on which that edition of
SQL Server software is installed. The only exception is a Passive machine in an Active-Passive cluster. Each client
needing access to any of the SQL Servers needs a Client Access License (CAL). You can think of a client as any
device (for example, a PC, workstation, terminal, or pager) utilizing the services of either SQL Server 2000 Standard
Edition or SQL Server 2000 Enterprise Edition. A client with a per-seat CAL can access any instance of SQL Server
in the environment. You can deploy additional installations of SQL Server with minimal expense, and you don't have
to buy additional client licenses. You simply acquire as many additional server licenses as you need, and your users
are already licensed to use them. Figure 4-1 illustrates a scenario requiring two SQL Server licenses and three CALs
deployed in the per-seat mode.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 4-1. A scenario requiring two Server Licenses and three CALs.

You should consider you licensing agreement carefully before installation as the licensing mode cannot be changed
without reinstalling SQL Server. In addition to the two licensing modes discussed above, there are some additional
considerations.

Multiplexing: Use of Middleware, Transaction

Servers, and Multitiered Architectures
Sometimes organizations develop network scenarios that use various forms of hardware and software that reduce the
number of devices that directly access or utilize the software on a particular server, often called multiplexing or
pooling hardware or software. For example, let's say a client PC is using a server application that calls the Microsoft
Transaction Service (MTS) component of Microsoft Windows 2000 Server on one server, which in turn pulls data
from a Microsoft SQL Server database on another server. In this case, the only direct connection to Microsoft SQL
Server comes from the server running MTS. The client PC has a direct connection to the server running MTS, but
also has an indirect connection to SQL Server because it is ultimately retrieving and using the SQL Server data
through MTS. Use of such multiplexing or pooling hardware and software does not reduce the number of CALs
required to access or utilize SQL Server software. A CAL is required for each distinct input to the multiplexing or
pooling software or hardware front end. If, in the preceding example, 50 PCs were connected to the MTS server, 50
SQL Server CALs would be required. This is true no matter how many tiers of hardware or software exist between
the SQL Server and the client devices that ultimately use its data, services, or functionality.

Multiple Instances
SQL Server 2000 includes a multi-instancing feature that allows customers to install SQL Server more than once on
a server. If you're using SQL Server Standard Edition, you must fully license each instance server (whether in per-seat
or per-processor mode). If you're using SQL Server Enterprise Edition, you can install an unlimited number of
instances on each machine without requiring any additional licensing.

There are additional issues when determining licensing requirements for SQL Server Personal Edition and for SQL
Server Desktop Engine. And if you are running SQL Server on MSCS, there are different licensing issues depending
on whether the SQL Servers are configured on the cluster as active-active or as active-passive. The Microsoft Web
This document is created with the unregistered version of CHM2PDF Pilot

site has full details on the licensing requirements in these, and all other, situations. Please refer to
http://www.microsoft.com/sql for any of the following:
 End User License Agreement
 Upgrade roadmaps to SQL Server 2000
 Additional Questions and Answers
This document is created with the unregistered version of CHM2PDF Pilot

34

Network Protocols
If you accept the default installation option during setup, SQL Server on Windows NT and Windows 2000 installs
named pipes, shared memory, and TCP/IP sockets as its interprocess communication (IPC) mechanisms for
communication with clients. SQL Server can simultaneously use many IPC mechanisms and networking protocols, but
any specific client connection uses only one for its connection. You learned in Chapter 3 that each SQL Server
networking interface is known as a Net-Library and represents a specific IPC mechanism. In addition to the Named
Pipes, Shared Memory, and TCP/IP Sockets Net-Libraries, you can install one or more of the following:
 Multiprotocol
 NWLink IPX/SPX
 AppleTalk ADSP
 Banyan VINES

Although the Named Pipes Net-Library remains a good choice, its use as a default is mostly historical. Named pipes
was the first, and for a while the only, IPC mechanism used by the early versions of SQL Server. Later, even when
TCP/IP sockets and IPX/SPX were supported, those protocols were more difficult to configure than named pipes,
requiring configuration at each client. For example, using TCP/IP required that the administrator configure an arcane
IP address at every client workstation for every installation of SQL Server it might access. Now, with the
sophisticated network naming services provided by Windows NT and Windows 2000 (such as WINS, DHCP, and
DNS), TCP/IP sockets are almost as easy to use as named pipes.

In addition, SQL Server 2000 allows the Shared Memory Net-Library to be used to communicate between SQL
Server and local clients. The Network Libraries dialog box that appears during a custom installation of SQL Server
2000 does not give you the option of specifying or not specifying this library. SQL Server always enables it. You can
determine whether you actually want shared memory to be used by local clients by using the Client Network Utility.
Figure 4-2 shows the dialog box for determining which protocols will be used by a client and in which order the
protocols will be tried. A check box at the lower left allows you to specify whether to use shared memory when
communicating with local SQL Server instances.

Figure 4-2. Using the Client Network Utility to determine which protocols the local clients will use.

Unless you have a compelling reason to choose a different network interface (the most compelling, of course, is if
your existing network or network standards dictate some other network choice), use the defaults because they
provide the most functionality. They are protocol-independent and allow the use of Windows Authentication Mode
when you connect to SQL Server; you can thus provide a single logon name to your users so they don't have to log
This document is created with the unregistered version of CHM2PDF Pilot

on to both the Windows NT or Windows 2000 domain and SQL Server. Windows Authentication Mode is an
important feature for convenience and ease of administration as well as for making your system more secure.
Windows Authentication Mode is also assumed with SQL Server replication services and when Performance Monitor
connects to SQL Server.

The Multiprotocol Net-Library is built using the operating system's remote procedure call (RPC) services. Prior to
SQL Server 2000, it was one of the only ways to get encryption over your network connection. Now, you should use
Multiprotocol only for compatibility with existing systems. Secure Sockets Layer (SSL) encryption can be enabled
over any Net-Library in SQL Server 2000 using the Server Network Utility. Multiprotocol is not supported on
machines running Windows 98 or for named instances of SQL Server.

Microsoft internal testing has found the TCP/IP Sockets Net-Library to be the fastest networking choice, although in
a typical LAN you rarely see the network as a performance bottleneck. In a low-speed WAN, however, this can be
an important issue. Some network administrators have also been concerned about potentially routing NetBIOS traffic
across their LANs and WANs. However, if SQL Server is not using the Named Pipes Net-Library and the
Multiprotocol Net-Library is not using named pipes under the covers for IPC, SQL Server is not using NetBIOS at
all and this is not a concern.

The NWLink IPX/SPX Net-Library is of most interest to those running Novell networking software on their clients
that access SQL Server. If you are using a Novell NetWare-based network and file server but your SQL Server
clients use Windows 98 or Windows NT and Windows 2000, using the NWLink IPX/SPX Net-Library is
unnecessary. If your clients use networking software provided by Novell, NWLink IPX/SPX is probably your best
choice. Server enumeration is available with NWLink IPX/SPX using the NetWare Bindery services.

Choose Banyan VINES if you interoperate with that environment. The Banyan VINES Net-Library uses StreetTalk
naming services for server enumeration and name resolution. Use the AppleTalk ADSP Net-Library if you will
support Apple Macintosh clients (using the Inprise—formerly Visigenic—ODBC driver for Macintosh) that run only
AppleTalk, not TCP/IP.

During installation, you must supply additional information for any of the network options you have selected—for
example, the port number or network name on which the server running SQL Server will "listen" for new connections
or broadcast its existence to a network naming service. In most cases, you should accept the default unless you have
a compelling reason not to. In the case of TCP/IP sockets, you should accept the default port number of 1433 for the
default instance only. This number is reserved for use with SQL Server by the Internet Assigned Numbers Authority
(IANA), and as such it should not conflict with a port used by any other server application on your computer. (This
assumes that the developers of other applications also follow the proper protocol of getting assigned numbers.)
Additional named instances of SQL Server installed on the same machine will need to use a different port number,
which can be dynamically assigned. We'll discuss named instances in detail later in this chapter.

SQL Server on Windows 98 doesn't support the server Named Pipes, Multiprotocol, Banyan VINES, and
AppleTalk Net-Libraries. SQL Server does support the client side of Named Pipes and Banyan VINES
Net-Libraries on Windows 98, so Windows 98 clients can use them to connect to SQL Server installations on
Windows NT and Windows 2000.
TIP
If you're new to networking and don't know your IP from your DHCP, don't fret. Accept the defaults, and configure
your networking later as your understanding improves (or get your network administrator to help you). Although it's a
good idea to understand your networking choices before installing SQL Server, you can easily change the networking
options later without disturbing your SQL Server environment.
This document is created with the unregistered version of CHM2PDF Pilot

34

Collation
The term collation refers to a set of rules that determine how data is compared and sorted. Character data is sorted
using rules that define the correct sequence of characters and options that specify case-sensitivity, accent-sensitivity,
and kana/width-sensitivity. During installation, you must decide which collation will be the default for your server. The
value you choose determines the default code page for non-Unicode characters, the default sort order for
non-Unicode characters, and the Unicode collation for your server. The collation you choose during setup determines
the collation of the system databases (master, model, tempdb, msdb, and distribution).

Earlier versions of SQL Server required that you make three separate decisions regarding collation during the
installation process. These decisions applied to all the data stored in all the databases on a single SQL Server and
were irrevocable. To change your decision, you had to rebuild your server and completely reload all your data. The
three decisions were:
 The character set to use for non-Unicode characters, which was determined by the code page
 The sort order to use for non-Unicode characters, which included information on case sensitivity and accent
sensitivity
 The sort order to use for Unicode characters, which was called the "Unicode Collation"

For SQL Server 2000, you must make these same three decisions, but you need to choose only one collation value,
which includes all three of your choices. I'll talk about how to specify your choice after I discuss the issues with each
of the three decisions.

Character Sets
Most non-Unicode characters are stored in SQL Server as a single byte (8 bits), which means that 256 (28) different
characters can be represented. But the combined total of characters in all the world's languages is more than 256. So
if you don't want to store all your data as Unicode, which takes twice as much storage space, you must choose a
character set that contains all the characters (referred to as the repertoire) that you need to work with. For
installations in the Western Hemisphere and Western Europe, the ISO character set (also often referred to as
Windows Characters, ISO 8859-1, Latin-1, or ANSI) is the default and is compatible with the character set used by
all versions of Windows in those regions. (Technically, there is a slight difference between the Windows character set
and ISO 8859-1.) If you choose ISO, you might want to skip the rest of this section on character sets, but you should
still familiarize yourself with sort order issues.

You should also ensure that all your client workstations use a character set that is consistent with the characters used
by your installation of SQL Server. If, for example, the character ¥, the symbol for the Japanese yen, is entered by a
client application using the standard Windows character set, internally SQL Server stores a byte value (also known as
a code point) of 165 (0xA5). If an MS-DOS-based application using code page 437 retrieves that value, that
application displays the character Ñ. (MS-DOS uses the term code pages to mean character sets; you can think of
the terms as interchangeable.) In both cases, the internal value of 165 is stored, but the Windows character set and
the MS-DOS code page 437 render it differently. You must consider whether the ordering of characters is what is
semantically expected (discussed later in the "Sort Orders" section) and whether the character is rendered on the
application monitor (or other output device) as expected.

SQL Server provides services in the OLE DB provider and the ODBC driver that use Windows services to perform
character set conversions. Conversions cannot always be exact, however, because by definition each character set
has a somewhat different repertoire of characters. For example, there is no exact match in code page 437 for the
Windows character Õ, so the conversion must give a close, but different, character.
This document is created with the unregistered version of CHM2PDF Pilot

Windows NT and Windows 2000 and SQL Server 2000 also support 2-byte characters that allow representation of
virtually every character used in any language. This is known as Unicode, and it provides many benefits—with some
costs. The principal cost is that 2 bytes instead of 1 are needed to store a character. SQL Server allows storage of
Unicode data by using the three datatypes nchar, nvarchar, and ntext. I'll discuss these in more detail when I talk
about datatypes in Chapter 7. The use of Unicode characters for certain data does not affect the character set chosen
for storing the non-Unicode data, which is stored using the datatypes char, varchar, and text.
The ASCII Character Set
It is worth pointing out that the first 128 characters are the same for the character sets ISO, code page 437, and
code page 850. These 128 characters make up the ASCII character set. (Standard ASCII is only a 7-bit character
set. ASCII was simple and efficient to use in telecommunications because the character and a "stop bit" for
synchronization could all be expressed in a single byte.) If your application uses ASCII characters but not the
so-called extended characters (typically characters with diacritical marks, such as à, Å, and ä) that differentiate the
upper 128 characters among these three character sets, it probably doesn't matter which character set you choose. In
this situation (only), whether you choose one of these three character sets or use different character sets on your client
and server machines doesn't matter because the rendering and sorting of every important character uses the same byte
value in all cases.

Earlier versions of SQL Server did not support Unicode, but they did support double-byte character sets (DBCS).
DBCS is a hybrid approach and is the most common way for applications to support Asian languages such as
Japanese and Chinese. With DBCS encoding, some characters are 1 byte and others are 2 bytes. The first bit in the
character indicates whether the character is a 1-byte or a 2-byte character. (In non-Unicode datatypes, each
character is considered 1 byte for storage; in Unicode datatypes, every character is 2 bytes.) To store two DBCS
characters, a field would need to be declared as char(4) instead of char(2). SQL Server correctly parses and
understands DBCS characters in its string functions. The DBCSs are still available for backward compatibility, but for
new applications that need more flexibility in character representation, you should consider using the Unicode
datatypes exclusively.

Sort Orders
The specific character set used might not be important in many sites where most of the available character sets can
represent the full range of English and Western European characters. However, in nearly every site, whether you
realize it or not, the basics of sort order (which is more properly called collating sequence) is important. Sort order
determines how characters compare and assign their values. It determines whether your SQL operations are case
sensitive. For example, is an uppercase A considered identical to a lowercase a? If your sort order is case sensitive,
these two characters are not equivalent, and SQL Server will sort A before a because the byte value for A is less than
that for a. If your sort order is case insensitive, whether A sorts before or after a is unpredictable unless you also
specify a preference value. Uppercase preference can be specified only with a case-insensitive sort order, and it
means that although A and a are treated as equivalent for comparisons, A is sorted before a.

If your data will include extended characters, you must also decide whether your data will be accent insensitive. For
our purposes, accents refer to any diacritical marks. So an accent-insensitive sort order treats a and ä as equivalent.

For certain character sets, you also need to specify two other options. Double-byte character sets can specify width
insensitivity, which means that equivalent characters represented in either 1 or 2 bytes are treated the same. For a
Japanese sort order, you can specify kana insensitive, which means that katakana characters are always unequal to
hiragana characters. If you use only ASCII characters and no extended characters, you should simply decide whether
you need case sensitivity and choose your collation settings accordingly.

Sort order affects not only the ordering of a result set but also which rows of data qualify for that result set. If a
query's search condition is
WHERE name='SMITH'
This document is created with the unregistered version of CHM2PDF Pilot

the case sensitivity determines whether a row with the name Smith qualifies.

Character matching, string functions, and aggregate functions (MIN, MAX, COUNT (DISTINCT), GROUP BY,
UNION, CUBE, LIKE, and ORDER BY) all behave differently with character data depending on the sort order
specified.

Sort order semantics

You can think of characters as having primary, secondary, and tertiary sort values, which are different for different
sort order choices. A character's primary sort value is used to distinguish it from other characters, without regard to
case and diacritical marks. It is essentially an optimization: if two characters do not have the same primary sort value,
they cannot be considered identical for either comparison or sorting and there is no reason to look further. The
secondary sort value distinguishes two characters that share the same primary value. If the characters share the same
primary and secondary values (for example A = a), they are treated as identical for comparisons. The tertiary value
allows the characters to compare identically but sort differently. Hence, based on A = a, apple and Apple are
considered equal. However, Apple sorts before apple when a sort order with uppercase preference is used. If there
is no uppercase preference, whether Apple or apple sorts first is simply a random event based on the order in which
the data is encountered when retrieved.

For example, a sort order specification of "case insensitive, accent sensitive, uppercase preference" defines all A-like
values as having the same primary sort value. A and a not only have the same primary sort value, they also have the
same secondary sort value because they are defined as equal (case insensitive). The character à has the same primary
value as A and a but is not declared equal to them (accent sensitive), so à has a different secondary value. And
although A and a have the same primary and secondary sort values, with an uppercase preference sort order, each
has a different tertiary sort value, which allows it to compare identically but sort differently. If the sort order didn't
have uppercase preference, A and a would have the same primary, secondary, and tertiary sort values.

Some sort order choices allow for accent insensitivity. This means that extended characters with diacritics are defined
with primary and secondary values equivalent to those without. If you want a search of name ='Jose' to find both
Jose and José, you should choose accent insensitivity. Such a sort order defines all E-like characters as equal:
E=e=è=É=é=ê=ë

NOTE
When deciding on the case sensitivity for your SQL Server databases, you should be aware that the case sensitivity
applies to object names as well as user data because object names (the metadata) are stored in tables just like user
data. So if you have a case-insensitive sort order, the table CustomerList is seen as identical to a table called
CUSTOMERLIST and to one called customerlist. If your server is case sensitive and you have a table called
CustomerList, SQL Server will not find the table if you refer to it as customerlist.

Binary sorting
As an alternative to specifying case or accent sensitivity, you can choose a binary sorting option, in which characters
are sorted based on their internal byte representation. If you look at a chart for the character set, you see the
characters ordered by this numeric value. In a binary sort, the characters sort according to their position by value, just
as they do in the chart. Hence, by definition, a binary sort is always case sensitive and accent sensitive and every
character has a unique byte value.

Binary sorting is the fastest sorting option because all that is required internally is a simple byte-by-byte comparison
of the values. But if you use extended characters, binary sorting is not always semantically desirable. Characters that
are A-like, such as Ä, ä, Å, and å, all sort after Z because the extended character A's are in the top 128 characters
and Z is a standard ASCII character in the lower 128. If you deal with only ASCII characters (or otherwise don't
care about the sort order of extended characters), you want case sensitivity, and you don't care about "dictionary"
sorting, binary sorting is an ideal choice. In case-sensitive dictionary sorting, the letters abcxyzABCXYZ sort as
This document is created with the unregistered version of CHM2PDF Pilot

AaBbCcXxYyZz. In binary sorting, all uppercase letters appear before any lowercase letters—for example,
ABCXYZabcxyz.

Specifying a collation
During the installation of SQL Server 2000, you are asked to specify a default collation for your server. This is the
collation that your system databases will use, and any new database you create will use this collation by default. In
Chapter 5, I'll describe how you can create a database with a different collation than the one chosen during
installation. Two types of collation are available; the documentation calls them SQL Collation and Windows Collation.
Figure 4-3 shows the installation dialog box in which you choose your collation. The two choices are described during
installation as Windows Locale Collation Designator and SQL Collation.

Figure 4-3. The dialog box for selecting the default collation.

SQL Collation is intended for backward compatibility and basically addresses all three collation issues in a single
description string: the code page for non-Unicode data, the sort order for non-Unicode data, and the sort order for
Unicode data. Figure 4-3 shows the default for an installation on a US English SQL Server, and the description of the
SQL Collation is "Dictionary Order, Case-Insensitive, For Use With 1252 Character Set". The code page for
non-Unicode data is 1252, the sort order for non-Unicode data is case insensitive, and the sort order for Unicode is
Dictionary Order. The actual name of this collation is SQL_Latin1_General_CP1_CI_AS, and you can use this
name when referring to this collation using SQL statements, as we'll see in later chapters. The prefix SQL indicates
that this is a SQL Collation as opposed to a Windows Collation. The Latin1_General part indicates that the Unicode
sort order corresponds to dictionary order, CP1 indicates the default code page 1252, and CI_AS indicates case
insensitive, accent sensitive. Other collation names use the full code page number; only 1252 has the special
abbreviation CP1. For example, SQL_Scandinavian_CP850_CS_AS is the SQL Collation using Scandinavian sort
order for Unicode data, code page 850 for non-Unicode data, and a case-sensitive, accent-sensitive sort order.

The SQL Collations are intended only for backward compatibility with previous SQL Server versions. There is a
fixed list of possible values for SQL Collations, and not every combination of sort orders, Unicode collations, and
code pages is possible. Some combinations don't make any sense, and others are omitted just to keep the list of
possible values manageable.

As an alternative to a SQL Collation for compatibility with a previous installation, you can define a collation based on
a Windows locale. In Windows Collation, the Unicode data types always have the same sorting behavior as
This document is created with the unregistered version of CHM2PDF Pilot

non-Unicode data types. A Windows Collation is the default for non-English SQL Server installations, with the
specific default value for the collation taken from your Windows operating system configuration. You can choose to
always sort by byte values for a binary sort, or you can choose case sensitivity and accent sensitivity. When using
SQL code to specify a Windows collation for a database, a column, or an expression, you can also specify a case
preference for sorts, and, where applicable, the sensitivity of the collation to width and kana. I'll talk about the SQL
statements for specifying collations when I discuss character datatypes in Chapter 7.

Performance Considerations
Binary sorting uses significantly fewer CPU instructions than sort orders with defined sort values. So binary sorting is
ideal if it fulfills your semantic needs. However, you won't pay a noticeable penalty for using either a simple
case-insensitive sort order (for example, Dictionary Order, Case Insensitive) or a case-sensitive choice that offers
better support than binary sorting for extended characters. Most large sorts in SQL Server tend to be I/O-bound, not
CPU-bound, so the fact that fewer CPU instructions are used by binary sorting doesn't typically translate to a
significant performance difference. When you sort a small amount of data that is not I/O-bound, the difference is
minimal; the sort will be fast in both cases.

A more significant performance difference results if you choose a sort order of Case Insensitive, Uppercase
Preference. Recall that this choice considers all values as equal for comparisons, which also includes indexing.
Characters retain a unique tertiary sort order, however, so they might be treated differently by an ORDER BY clause.
Uppercase Preference, therefore, can often require an additional sort operation in queries.

Consider a query that specifies WHERE last_name >= 'Jackson' ORDER BY last_name. If an index exists on the
last_name field, the query optimizer will likely use it to find rows whose last_name value is greater than or equal to
Jackson. If there is no need to use Uppercase Preference, the optimizer knows that because it is retrieving records
based on the order of last_name, there is also no need to physically sort the rows because they were extracted in
that order already. Jackson, jackson, and JACKSON are all qualifying rows. All are indexed and treated identically,
and they simply appear in the result set in the order in which they were encountered. If Uppercase Preference is
required for sorting, a subsequent sort of the qualifying rows is required to differentiate the rows even though the index
can still be used for selecting qualifying rows. If many qualifying rows are present, the performance difference between
the two cases (one needs an additional sort operation) can be dramatic. This doesn't mean that you shouldn't choose
Uppercase Preference. If you require those semantics, performance might well be a secondary concern. But you
should be aware of the trade-off and decide which is most important to you.

Another performance issue arises when you decide between SQL Collation and Windows Collation. Because any
character data defined using a Windows Collation is sorted based on the Unicode sort table for the collation, all
non-Unicode data must be converted to Unicode during the sort. I ran some simple tests with two 10,000-row tables.
The only difference between the tables was in a column called last_name. In one table, last_name was defined using
a SQL Collation, and in the other, it was defined using an equivalent Windows Collation. Repeated sorts of the
column in both tables showed a 15 to 30 percent increase in CPU time for the sort on the Unicode column.
This document is created with the unregistered version of CHM2PDF Pilot

34

Multiple Instances
SQL Server 2000 allows you to install multiple instances of SQL Server on a single computer. One instance of SQL
Server can be referred to as the default instance, and all others must have names and are referred to as named
instances. Each instance is installed separately and has its own default collation, its own system administrator
password, and its own set of databases and valid users.

The ability to install multiple instances on a single machine provides benefits in several different areas. First is the area
of application hosting. One site can provide machine resources and administration services for multiple companies.
These might be small companies that don't have the time or resources (human or machine) to manage and maintain
their own SQL Servers. They might want to hire another company to "host" their SQL Server private or public
applications. Each company will need full administrative privileges over its own data. However, if each company's
data were stored in a separate database on a single SQL Server, any SQL Server system administrator (sa) would be
able to access any of the databases. Installing each company's data on a separate SQL Server allows each company
to have full sa privileges for its own data but no access at all to any other company's data. In fact, each company
might be totally unaware that other companies are using the same application server.

A second area in which multiple instances provide great benefits is server consolidation. Instead of having 10
separate machines to run 10 separate applications within a company, all the applications can be combined on one
machine. With separate instances, each application can still have its own administrator and its own set of users and
permissions. For the company, this means fewer boxes to administer and fewer operating system licenses required.
For example, one Windows NT 4 Server, Enterprise Edition (or Windows 2000 Advanced Server) with 10 SQL
Server instances costs less than 10 separate server boxes.

An added benefit to running multiple instances on one big computer is that you are more likely to take care of a more
powerful computer, which will lead to greater reliability. For example, if you decide to purchase an eight-way
Compaq ProLiant with 4 GB of RAM, it might cost you about $100,000. With that kind of investment, you'll
probably want to set it up in its own climate-controlled data center instead of next to someone's desk. You'll
meticulously maintain both the hardware and the software and only let trained engineers get near it. This means that
reliability and availability will be greatly enhanced.

The third area in which multiple instances are beneficial is testing and support. Because the separate instances can be
different versions of SQL Server or even the same version with different service packs installed, you can use one box
for reproducing problem reports or testing bug fixes. You can verify which versions the problems can be reproduced
on and which versions they can't. A support center can similarly use multiple instances to make sure that it can use the
same exact version that the customer has installed.

Installing Named Instances

When you install multiple SQL Server instances on a single machine, only one instance is the "default" instance. This is
the SQL Server that is accessed in the same way that SQL Servers in previous versions were accessed—by
supplying the machine name as the name of the SQL Server. For example, to use the Query Analyzer to access the
SQL Server on my machine called KALESSIN, I just type KALESSIN in the SQL Server text box in the Connect
To SQL Server dialog box. Any other instances on the same machine will have an additional name, which must be
specified along with the machine name. I can install a second SQL Server instance with its own name—for example,
SQL2000. Then I must enter KALESSIN\SQL2000 in the initial text box to connect using Query Analyzer.

Your default instance can be SQL Server 6.5, SQL Server 7, or SQL Server 2000. In fact, if you're using the
version switch capability (discussed shortly) to switch between SQL Server 6.5 and SQL Server 7 or between SQL
This document is created with the unregistered version of CHM2PDF Pilot

Server 6.5 and SQL Server 2000, your default instance can be this switched instance. Your default instance cannot
be SQL Server 6 even if it is being version-switched with a later version. If you have SQL Server 6 or an older SQL
Server 4.2 installation on your machine, you cannot install SQL Server 2000 at all.

Named instances can only be SQL Server 2000. Microsoft documents that you can have a maximum of 16 instances
on a single machine, but this is only the supported limit. There is nothing hardcoded into the system to prevent you
from installing additional instances, as long as you have the resources on the machine to support them.

Each instance has its own separate directory for storing the server executables, and each instance's data files can be
placed wherever you choose. During installation, you can specify the desired path for the program files and the data
files. Each instance also has its own SQL Server Agent service. The service names for the default instance do not
change—they are MSSQLServer and SQLServerAgent. For an instance named SQL2000, the services are named
MSSQL$SQL2000 and SQLAGENT$SQL2000.

However, only one installation of the tools is used for all instances of SQL Server 7 and SQL Server 2000.
However, you will still have your SQL Server 7 Books Online, as well as your SQL Server 7 Server Network Utility
available. In addition, there is only one Search Service, one Distributed Transaction Coordinator (DTC) service, one
copy of English Query, one copy of the development libraries, and one copy of Microsoft SQL Server 2000 Analysis
Services (formerly OLAP Services).

Named Instance Server Connectivity

I mentioned earlier in this chapter that the default Net-Libraries for SQL Server 2000 are Named Pipes, Shared
Memory, and TCP/IP. Of the additional protocols that can be added, Multiprotocol, Banyan VINES, and AppleTalk
are not supported for named instances. The default instance listens on a pipe named \\.\pipe\sql\query and on TCP
port number 1433. A named instance listens on a pipe named \\.\pipe\MSSQL$<instance name>\sql\query, so for my
SQL2000 instance, it is \\.\pipe\MSSQL$SQL2000\sql\query.

Determining which TCP port to use for a named instance is a little more problematic. Microsoft reserved port
number 1433 but was unable to reserve additional ports. During setup, you can choose any port number, but you
should be aware that other services from other vendors might conflict with the port number you choose. To avoid
problems, the setup program allows you to specify 0 for the port number, which means that every time the SQL
Server starts, it finds an unused port number to use. You can see what port has been chosen by using the Server
Network Utility, selecting the name of the instance, and then displaying the properties for TCP/IP.

Allowing multiple instances of SQL Server to run on a single machine requires a Listener service to determine what
instances are actually running on a machine. The new Listener service runs on UDP port 1434 but doesn't have an
actual service name on the machine. This service exposes which SQL Server instances are installed, their network
addresses, and so forth, for use by SQL Server client tools. At server startup, a component of every SQL Server
instance checks whether anyone is listening on UDP port 1434. If no other instance is listening, it starts listening and
becomes the listener service for the machine.

When a client asks to connect to a machine, the tool being used communicates with the Listener service server,
asking for information about all the instances installed, their addresses, whether they are a cluster, and so forth. For
each instance, the Listener service tells the client computer which server Net-Libraries and network addresses the
instance is listening on. After the client computer receives this report, it chooses a Net-Library that is enabled on both
the application computer and on the instance of SQL Server, and it makes a connection to the address listed for that
Net-Library in the packet.
CAUTION
Because you can only have one set of client tools installed on a machine, as soon as you install an instance of SQL
Server 2000, you lose any SQL Server 7 tools you had on your machine, except for Books Online and the Server
Network Utility. If one of the reasons you're using multiple instances is to have multiple versions available for testing
purposes, you will only be able to test the SQL Servers. If you're doing troubleshooting and think the problem might
This document is created with the unregistered version of CHM2PDF Pilot

be with the tools themselves, you're out of luck. You can't test using a different version of the tools on the same
machine.
This document is created with the unregistered version of CHM2PDF Pilot

34

Installing SQL Server

Now that you understand all the preliminary considerations of installing and using SQL Server, you're ready to install
the software. The actual mechanics of installation are simple—the installation program starts automatically when you
insert the CD into the drive; if it doesn't, you can initiate the installation by executing the autorun.exe program at the
root of the SQL Server CD.

SQL Server 2000 is installed using the InstallShield program, which is common to many Microsoft products. One
advantage of using InstallShield instead of a specific SQL Server Setup program is that you can use the Add/Remove
Programs options in Control Panel to add SQL Server components or to completely remove a SQL Server instance.
In addition, InstallShield makes the process of running an unattended installation much easier than in previous versions.

The lengthiest part of installation, along with the mechanics of copying the files, is building and configuring SQL
Server's master database, which stores configuration information and information about all other databases as well as
many system stored procedures. If you accept the default collation, the installation program simply copies a mostly
prebuilt master database from the files master.mdf and mastlog.ldf on the CD. These files are in a subfolder called
Data in the x86 folder. Then the installation program does a small amount of additional configuration. A new
installation using the default character set can typically take less than 15 minutes from start to finish (sometimes as little
as 5 minutes, depending on the speed of your hardware and how much "think time" you need for the questions). If you
do not accept the default collation, the installation program must reindex the entire master database. This reindexing is
a totally automatic operation, but it can add 5 to 10 minutes to the installation time. The total installation time is still
usually under 30 minutes.

Upgrading from a Previous Version

The installation program will detect if a version of SQL Server is already installed on your machine. Depending on
whether the previous installation is SQL Server 6 or SQL Server 7, you will be presented with some choices to make
when installing SQL Server 2000.

Upgrading from SQL Server 7

If your machine has SQL Server 7 installed, you will be given a choice of installing a new, named instance or
upgrading the existing SQL Server 7 to SQL Server 2000. If you choose to upgrade the previous installation, the
installation program rebuilds all the system stored procedures to ensure that the most recent versions are installed. The
database files for each database are modified slightly to conform to the structure of SQL Server 2000 database files
(which I'll discuss in detail in Chapter 5). The actual table data isn't touched, and after the upgrade is complete, all
your SQL Server 7 databases will be available on your SQL Server 2000 server.

Alternatively, you can upgrade an individual database from SQL Server 7 to SQL Server 2000 if you install a
separate named instance of SQL Server 2000 alongside your original SQL Server 7 server. You can load database
backups from SQL Server 7 into SQL Server 2000, you can use the sp_attach_db procedure to connect database
files from a SQL Server 7 database to your SQL Server 2000, or you can use a feature called the Copy Database
Wizard. This wizard is based on the sp_attach_db functionality but is available as a GUI wizard to make the process
a bit more painless. Moving a database from SQL Server 7 to SQL Server 2000 is a one-way operation, partly
because of the change to the database files that takes place. You cannot load a SQL Server 2000 backup into a SQL
Server 7 database, and you cannot use sp_attach_db with SQL Server 7 to connect to files from a SQL Server
2000 database.
This document is created with the unregistered version of CHM2PDF Pilot

Upgrading from SQL Server 6.5

If the installation process detects that you have SQL Server 6.5 installed (and you don't have SQL Server 7 available
for version switching), you are given the choice of making the new SQL Server 2000 server a separate named
instance or setting up a version switch with your SQL Server 6.5 server. In either case, your SQL Server 6.5 is not
affected and the installation is treated as a new installation rather than as an upgrade. Because of tremendous
structural changes between versions 6.5 and 7, all your version 6.5 data must be completely rebuilt in order to be
used with SQL Server 2000. The rebuilding process is called a database upgrade and can be accomplished using
the SQL Server Upgrade Wizard, which is available from a program group called Microsoft SQL Server - Switch.
You can run this wizard while the installation of SQL Server 2000 is taking place, or you can run it at a later time. The
only way to accomplish the upgrade is by using the wizard.

If you choose not to install SQL Server 2000 as a separate named instance, your new SQL Server 2000 will be the
default instance. Since your SQL Server 6.5 can only be a default instance and you can only have one default instance
running, you must decide whether you want to run SQL Server 6.5 or the new SQL Server 2000 instance at any
given time. You can version-switch between the two by choosing Programs/Microsoft SQL Server - Switch from the
Start menu and then choosing the option to switch to the desired version. Only the default instance of SQL Server can
be involved in a version switch.

For more details on installation, see the online documentation and the whitepaper titled "Installing Microsoft SQL
Server 2000"on the companion CD.
This document is created with the unregistered version of CHM2PDF Pilot

34

Basic Configuration After

Installation
After the installation, you should verify the basic operation of SQL Server. SQL Server 2000 is mostly
self-configuring. Many options are available for fine-tuning the configuration, but these are necessary for specialized
purposes only. I'll discuss configuration in detail in Chapter 17. Initially, you should leave the default configuration
alone unless you have a particular reason to change it.

Starting the SQL Server Service

After a successful installation, start SQL Server. The most common way to do this is to use the SQL Server Service
Manager. From the Start menu, choose Programs, Microsoft SQL Server, and then Service Manager. The installation
program also puts the SQL Server Service Manager in the startup group for the operating system, so after you reboot
your machine, an icon for the SQL Server Service Manager will appear on the taskbar, in the corner near the clock.
You can also use the Services applet in Windows NT and Windows 2000. In Windows NT, you can get to the
applet from Control Panel. In Windows 2000, you choose Programs/Administrative Tools/Services from the Start
menu. From the Microsoft Management Console, you can then find the service name for any SQL Server instance in
the right pane. You can use SQL Enterprise Manager to start a SQL Server instance, or you can issue a NET
START MSSQLSERVER command from a Windows NT or Windows 2000 console (command prompt). If you're
starting a named instance, you must supply the instance name. You can also configure any instance of SQL Server to
start automatically, either by using the Services applet or by editing the properties of SQL Server in Enterprise
Manager.

After SQL Server is running, you can use one of the most basic applications, SQL Query Analyzer, to make sure
that you can connect. Initially, the only available SQL Server login name (sa) has a null password, so you can leave sa
in the Login Name text box and leave the Password text box blank. Or you can choose the Windows Authentication
option so that no login name or password is required. Then you change your database to the pubs sample database,
and run a couple of simple queries (for example, SELECT * FROM authors ORDER BY au_lname) to make sure
that SQL Server is running correctly.

Changing the System Administrator Password

During installation, you are given the option of setting up Windows Authentication Mode only or using Mixed Mode
security. If you choose Mixed Mode security, you are asked to specify a password for the SQL Server login sa. If
you want the password to be blank, you must specify this by using a check box, and the installation screen lets you
know that this is not recommended. If you did choose a blank password during installation, after you verify that SQL
Server is running and responding to queries, you should change the password for the sa account. From the Query
Analyzer, use sp_password to change the sa password. Be sure to pick a password you'll remember, because by
design there is no way to read a password (it is stored in encrypted form)—it can only be changed to something else.

Using Query Analyzer from the master database to change the password to Banks_14, you'd issue the following
command:
sp_password NULL, 'Banks_14', sa

Using Enterprise Manager, you can change the password from the server's Security folder. Choose Logins, and then
This document is created with the unregistered version of CHM2PDF Pilot

double-click sa in the right pane.

Note that the actual password is stored in a table as SQL Server data. So if you've chosen a case-sensitive sort
order, passwords will also be case sensitive and must be entered exactly as defined.
SQL Enterprise Manager vs. SQL Statements
You can do nearly any task with SQL Server by directly executing SQL Server statements (including stored
procedure calls) using tools such as SQL Query Analyzer. However, in some cases it is simpler to use Enterprise
Manager, which provides a front end for these commands and frees you from needing to know exact steps and
syntax. Administrators who are proficient with Microsoft Visual Basic for Applications (VBA) might choose to use
simple VBA scripts that use the SQL Server database administration object model, known as SQL-DMO (SQL
Distributed Management Objects), or the Windows Management Instrumentation (WMI) interface.

Since this book is geared toward database developers and administrators, I'll typically show the SQL statements and
stored procedures that are used to accomplish a task and simply reference the easier methods. I'll also tell you what
to do without providing step-by-step directions. You can consult the SQL Server documentation for the exact syntax
of all options. This book is meant to complement, but not replace, the product documentation.

Configuring SQL Server's Error Log

A new error log is created each time the SQL Server service is started. If you want to start a new error log without
stopping and restarting SQL Server, you can use the sp_cycle_errorlog system stored procedure. Keep in mind that
the new error log created when you execute this procedure will not contain all the normal boot messages that are put
in the error log when SQL Server starts up. You'll have to refer to an older copy of the error log to see these
messages. However, if you execute sp_cycle_errorlog too many times, you might not have any copies that contain
the boot messages. By default, SQL Server retains copies of the previous six error logs and gives the most recent
copy (prior to the currently active error log) the extension .1, the second most recent copy the extension .2, and so
on. The current error log has no extension. You can change the number of error logs maintained by editing the
Registry for any instance. For the default instance, find the key
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\MSSQLServer and edit it by adding a new
value. Define the value with the name NumErrorLogs, and the type REG_DWORD. Supply any initial value desired,
but keep in mind that the value you enter when editing the Registry will be in hexadecimal format.

For a named instance, you need to find the key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft

SQL Server\<SQL Server Instance Name>\MSSQLServer. Again, use the Edit feature of the Registry to add a new
value with the name NumErrorLogs of type REG_DWORD, and supply an initial value in hexadecimal format.

Working with Multiple Instances

As mentioned earlier, in order to connect to a named instance, you must specify both the server name and the
instance name. To use the OSQL command line utility to connect to an instance named SQL2000, you use this
command:
osql /U sa /S KALESSIN\SQL2000

SQL Server 2000 includes a property function called SERVERPROPERTY that allows you to programmatically find
out information about your SQL Server instance. SQL Server Books Online lists all the parameters of the function,
but the ones relevant to multiple instances are InstanceName, MachineName, and ServerName. I can run the
following three commands:
SELECT SERVERPROPERTY('InstanceName')
SELECT SERVERPROPERTY('MachineName')
SELECT SERVERPROPERTY('ServerName')
This document is created with the unregistered version of CHM2PDF Pilot

On my machine called RAKAM, with my instance named SQL2000, I get the following results:
SQL2000
RAKAM
RAKAM\SQL2000

Other parameters of SERVERPROPERTY allow you to determine which licensing mode you're running under, what
service pack you're using, and what the operating system process ID is. The latter can be very important if you need
to access one instance through the Windows Task Manager. The process image name will be sqlservr.exe for all
instances, but the column labeled PID (process ID) will allow you to distinguish between multiple instances.

As mentioned earlier, each SQL Server instance has its own path for the executable program files and its own default
location of data files. If you have more than a couple of instances, it can become quite a chore to remember the
location of each instance. Fortunately, there is a tool in the Windows NT 4 Resource Kit and the Windows 2000
Support Tools that can help you. You need to know the name of the Registry key where the information about each
instance is stored, but if you know that, you can use the utility called reg.exe to find out the file paths used. To find the
program file path for a named instance, you can type the following in a command window, all on a single line:
REG QUERY "HKLM\Software\Microsoft\Microsoft SQL Server\<InstanceName>\
Setup\SQLPath"

To find the default data file path, change SQLPath at the end of the key name to SQLDataRoot:
REG QUERY "HKLM\Software\Microsoft\Microsoft SQL Server\<InstanceName>\
Setup\SQLDataRoot"

The old DB-Library API doesn't work with named instances because the code that looks for your server name does
not recognize the Servername\InstanceName specification. Microsoft has been saying for years that DB-Library will
not continue to be supported, and now they're showing us that they meant it. This means that you cannot use the
ISQL command line tool to connect to a named instance. At least, by default you cannot connect, but as is frequently
the case, there's a way around it. For those of you who refuse to give up your DB-Library, you can use the Client
Network Utility to alias your named instance to a one-part server name that ISQL will recognize. When specifying the
properties of this alias, you must specify the TCP port number. If this instance has its port number dynamically
determined, you can check the box that says to dynamically determine the port for the alias as well, as shown in
Figure 4-4.

Figure 4-4. Using the Client Network Utility to alias a named instance of SQL Server.
This document is created with the unregistered version of CHM2PDF Pilot

34

Remote and Unattended

Installation
SQL Server can be installed on a remote computer, and it can be totally scripted, requiring no user intervention. With
SQL Server 2000, the InstallShield-based installation, remote setup, and unattended setup are almost the same thing.
The remote installation option can be useful when you are responsible for many SQL Server installations. In fact, the
feature was added (in the days predating Microsoft Systems Management Server) at the behest of Microsoft's own
MIS organization, which manages more than 200 machines running SQL Server (and thousands of databases).
Installing and upgrading those 200 servers remotely from one computer greatly simplified MIS's work.

Other sites script their installations by using the unattended installation feature to deploy many servers instead of using
the remote installation option, which is still an interactive process. Microsoft Systems Management Server uses this
capability and can be used to install SQL Server at multiple sites. (In fact, unattended installation scripts and a PDF
file, used by Systems Management Server, are provided on the root directory of the SQL Server CD.)

The unattended installation option is also useful if you're embedding SQL Server in a turnkey solution and want to
provide a single, unified installation. As should be apparent by now, the SQL Server installation program makes
multiple entries to the Registry, changes system paths and various system settings, creates program groups and user
accounts, and performs basic initial configuration for its operation. The installation program is far from a glorified
file-copy program. It would not be realistic for solution providers to create their own programs to install SQL Server
as part of their services or products. But they can easily achieve the same results by simply creating a script that can
drive the SQL Server installation program without any user intervention.

Remote Installation
Remote installation is similar to a normal, local installation. After you choose the remote installation option (by clicking
Remote in the first dialog box of the installation program), you are prompted to answer a few questions in addition to
those a local installation needs. Along with providing the machine name of the anticipated SQL Server, you must do
the following:
 Run the installation program from an account with administrator privileges on the remote (target) server
 Be aware that drive paths should be relative to the remote server

After specifying that you want a remote installation, the information that you provide in the rest of the dialog boxes is
collected into a file called setup.iss. Then a program is started on the remote computer, and the local installation
program exits. The remote program (remsetup.exe) copies files to the \admin$ share directory and runs an unattended
installation on the remote computer using the options specified in the setup.iss file.

Unattended Installation
Unattended installation creates a file using the options you selected in the SQL Server installation program. To invoke
an unattended installation, you must first generate an InstallShield ISS file using one of the three methods described in
the following list. All the methods require that you run the SQL Server installation program from a command prompt
instead of using the autorun.exe program. The program is in the x86\Setup folder on the SQL Server installation CD.
 Normal Installation The normal installation creates a file called setup.iss. This file is saved in the Install
folder in the MSSQL7 directory (for the default instance) or in program directory for the named instance. The
setup.iss file contains your answers to all the questions in the installation dialog boxes.
This document is created with the unregistered version of CHM2PDF Pilot

 Advanced Installation During installation, you are presented with a dialog box that allows you to choose to
create a named instance, upgrade an existing instance (if one exists), or choose advanced options. If you
choose Advanced, you can choose the option to record an Unattended ISS file. You then go through the rest
of the dialog boxes, providing answers to questions, and your answers are saved in a setup.iss file in the root
directory of your operating system. SQL Server isn't actually installed.
 Manual File Creation The installation CD contains a sample setup file called sqlins.iss. You can use a text
editor to modify this file to correspond to the installation options you want. For details of the format of this file,
see the online documentation.

After the setup ISS file is created, you can move or copy it to another location for use on other servers. For
subsequent automated installations, start Setupsql and specify a previously generated setup ISS file as input by using
the -f1 command-line option. The syntax for this command is:
START Setupsql.exe -f1 <full path to iss file> -SMS -s

The -s switch causes setup to run in a silent mode with no user interface. If the -SMS switch is not specified, the setup
program launches InstallShield to perform the installation and then exits. Control immediately returns to the user. If
the -SMS switch is specified, the setup program waits until the installation is complete before terminating. This allows
an automated process or a user-created batch file to launch the setup program, wait for setup to complete, and then
perform additional actions. Without the -SMS switch, the additional actions might be initiated before the installation of
SQL Server completes.
NOTE
For remote or unattended installation, you might choose to copy all the files from the CD to a network share.
Normally, you should not notice any difference between network installation and installation from the CD.

Changing Installation Options

All the decisions you make during installation can be changed at a later time. As mentioned earlier, you can change
your decisions about the account under which the SQL Server service runs and whether SQL Server should start
automatically when the system starts. You do this either using the Services applet or by editing the properties of your
SQL Server through SQL Enterprise Manager. You can also change the Security options for SQL Server by using
SQL Enterprise Manager to specify whether SQL Server should use Windows Authentication only or also use
standard SQL Server Authentication. You can change the network interface options by using the Server Network
Utility. The collation cannot be changed on the system databases, but new databases can be built with any collation,
and you can change an existing user database to a new collation.

SQL Server 2000 provides the following additional utilities to carry out these special operations:
 Rebuilding the master database The Rebuild master utility rebuildm.exe in the Program Files\Microsoft
SQL Server\80\Tools\Binn directory rebuilds the master database completely. You might have to restore
backups of the msdb and model system databases after you rebuild the master database.
 Rebuilding the Registry entries You can use the setup program to restore the SQL Server Registry entries
if they become corrupted. The same Advanced Options dialog box that allows you to build an unattended
installation file gives you the option of rebuilding the Registry. You might have to rebuild the Registry if the
Windows NT and Windows 2000 Registry was partially corrupted (for example, if it was inadvertently
deleted using the Registry editing program) or if Windows NT and Windows 2000 was reinstalled (not
upgraded) on a machine and the new Registry is not aware of the previous SQL Server installation.

Adding Additional Components

If you want to add components, you can rerun the SQL Server installation program from the original CD. The
program detects and displays components that you already have installed and lets you select additional components.
The main dialog box is very specific about the actions it will perform: "Selecting components already installed will not
This document is created with the unregistered version of CHM2PDF Pilot

reinstall them and unselecting components already installed will not remove them." The components you might want to
install after the initial installation include:
 Server components You can install replication support, the performance counters, and full-text search
capabilities.
 Management tools You can install SQL Enterprise Manager, SQL Query Analyzer, SQL Server Profiler,
and DTC Client Support.

Two other tools are available with SQL Server 2000 that are not part of the standard installation. The initial
installation screen gives you the choice of installing SQL Server (Personal or one of the full server editions), SQL
Server 2000 Analysis Services, or English Query. If you choose either of the latter two options, SQL Server itself is
not installed and any existing SQL Server 2000 instances are not affected.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
SQL Server offers unprecedented ease of installation for a database server of its caliber. But you should understand
your choices in terms of hardware, licensing, and sort orders before you proceed too far down the path of a major
application rollout. Installing SQL Server is also fast and reliable. You need to do very little custom configuration
because SQL Server 2000 automatically configures itself as needed.

The SQL Server installation process offers options for unattended and remote installation, for installing multiple
instances of SQL Server on the same machine, for installing a Desktop version, and for client-tools-only installation.
Utilities are provided for changing many of the options chosen during the initial installation, but to add new
components, you must keep your original CD handy (if you haven't copied its contents to a network share).
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 5

Databases and Database Files

Simply put, a Microsoft SQL Server database is a collection of objects that hold and manipulate data. A typical SQL
Server installation has only a handful of databases, but it's not unusual for a single installation to contain several dozen
databases. (Theoretically, one SQL Server installation can have as many as 32,767 databases. But practically
speaking, this limit would never be reached.)

A SQL Server database:

 Is a collection of many objects, such as tables, views, stored procedures, and constraints. The theoretical
limit is 231 -1 (more than 2 billion) objects. Typically, the number of objects ranges from hundreds to tens of
thousands.
 Is owned by a single user account but can contain objects owned by other users.
 Has its own set of system tables that catalog the definition of the database.
 Maintains its own set of user accounts and security.
 Is the primary unit of recovery and maintains logical consistency among objects in the database. (For
example, primary and foreign key relationships always refer to other tables within the same database, not to
other databases.)
 Has its own transaction log and manages the transactions within the database.
 Can participate in two-phase commit transactions with other SQL Server databases on the same server or
different servers.
 Can span multiple disk drives and operating system files.
 Can range in size from 1 MB to a theoretical limit of 1,048,516 TB.
 Can grow and shrink, either automatically or by command.
 Can have objects joined in queries with objects from other databases in the same SQL Server installation or
on linked servers.
 Can have specific options set or disabled. (For example, you can set a database to be read-only or to be a
source of published data in replication.)
 Is conceptually similar to but richer than the ANSI SQL-schema concept (discussed later in this chapter).

A SQL Server database is not:

 Synonymous with an entire SQL Server installation.
 A single SQL Server table.
 A specific operating system file.

A database itself isn't synonymous with an operating system file, but a database always exists in two or more such
files. These files are known as SQL Server database files and are specified either at the time the database is created,
using the CREATE DATABASE command, or afterwards, using the ALTER DATABASE command.
This document is created with the unregistered version of CHM2PDF Pilot

34

Special System Databases

A new SQL Server 2000 installation automatically includes six databases: master, model, tempdb, pubs,
Northwind, and msdb.

master
The master database is composed of system tables that keep track of the server installation as a whole and all other
databases that are subsequently created. Although every database has a set of system catalogs that maintain
information about objects it contains, the master database has system catalogs that keep information about disk
space, file allocations, usage, systemwide configuration settings, login accounts, the existence of other databases, and
the existence of other SQL servers (for distributed operations). The master database is absolutely critical to your
system, so be sure to always keep a current backup copy of it. Operations such as creating another database,
changing configuration values, and modifying login accounts all make modifications to master, so after performing
such activities, you should back up master.

model
The model database is simply a template database. Every time you create a new database, SQL Server makes a
copy of model to form the basis of the new database. If you'd like every new database to start out with certain
objects or permissions, you can put them in model, and all new databases will inherit them.

tempdb
The temporary database, tempdb, is a workspace. SQL Server's tempdb database is unique among all other
databases because it's re-created—not recovered—every time SQL Server is restarted. It's used for temporary
tables explicitly created by users, for worktables to hold intermediate results created internally by SQL Server during
query processing and sorting, and for the materialization of static cursors and the keys of keyset cursors. Operations
within tempdb are logged so that transactions on temporary tables can be rolled back, but the records in the log
contain only enough information to roll back a transaction, not to recover (or redo) it. No recovery information is
needed because every time SQL Server is started, tempdb is completely re-created; any previous user-created
temporary objects (that is, all your tables and data) will be gone. Logging only enough information for rolling back
transactions in tempdb was a new feature in SQL Server 7 and can potentially increase the performance of INSERT
statements to make them up to four times faster than inserts in other (fully logged) databases.

All users have the privileges to create and use private and global temporary tables that reside in tempdb. (Private and
global table names have # and ## prefixes, respectively, which I'll discuss in more detail in Chapter 6.) However, by
default, users don't have the privileges to USE tempdb and then create a table there (unless the table name is
prefaced with # or ##). But you can easily add such privileges to model, from which tempdb is copied every time
SQL Server is restarted, or you can grant the privileges in an autostart procedure that runs each time SQL Server is
restarted. If you choose to add those privileges to the model database, you must remember to revoke them on any
other new databases that you subsequently create if you don't want them to appear there as well.

pubs
The pubs database is a sample database used extensively by much of the SQL Server documentation and in this
This document is created with the unregistered version of CHM2PDF Pilot

book. You can safely delete it if you like, but it consumes only 2 MB of space, so unless you're scrounging for a few
more megabytes of disk space, I recommend leaving it. This database is admittedly fairly simple, but that's a feature,
not a drawback. The pubs database provides good examples without a lot of peripheral issues to obscure the central
points. Another nice feature of pubs is that it's available to everyone in the SQL Server community, which makes it
easy to use to illustrate examples without requiring the audience to understand the underlying tables or install some
new database to try out your examples. As you become more skilled with SQL Server, chances are you'll find
yourself using this database in examples for developers or users.

You shouldn't worry about making modifications in the pubs database as you experiment with SQL Server features.
You can completely rebuild the pubs database from scratch by running a script in the \Install subdirectory (located
right under the SQL Server installation directory). In SQL Query Analyzer, open the file named Instpubs.sql and
execute it. You do need to make sure that there are no current connections to pubs, because the current pubs
database is dropped before the new one is created.

Northwind
The Northwind database is a sample database that was originally developed for use with Microsoft Access. Much
of the documentation dealing with APIs uses Northwind, as do some of the newer examples in the SQL Server
documentation. It's a bit more complex than pubs, and at almost 4 MB, slightly larger. I'll be using it in this book to
illustrate some concepts that aren't easily demonstrated using pubs. As with pubs, you can safely delete Northwind if
you like, although the disk space it takes up is extremely small compared to what you'll be using for your real data. I
recommend leaving Northwind there.

The Northwind database can be rebuilt just like the pubs database, by running a script located in the \Install
subdirectory. The file is called Instnwnd.sql.

msdb
The msdb database is used by the SQL Server Agent service, which performs scheduled activities such as backups
and replication tasks. In general, other than performing backups and maintenance on this database, you should ignore
msdb. (But you might take a peek at the backup history and other information kept there.) All the information in msdb
is accessible from the SQL Server Enterprise Manager tools, so you usually don't need to access these tables directly.
Think of the msdb tables as another form of system tables: just as you should never directly modify system tables, you
shouldn't directly add data to or delete data from tables in msdb unless you really know what you're doing or are
instructed to do so by a Microsoft SQL Server technical support engineer.
This document is created with the unregistered version of CHM2PDF Pilot

34

Database Files
A database file is nothing more than an operating system file. (In addition to database files, SQL Server also has
backup devices, which are logical devices that map to operating system files, to physical devices such as tape drives,
or even to named pipes. I won't be discussing files that are used to store backups.) A database spans at least two,
and possibly several, database files, and these files are specified when a database is created or altered. Every
database must span at least two files, one for the data (as well as indexes and allocation pages), and one for the
transaction log. SQL Server 2000 allows the following three types of database files:
 Primary data files Every database has one primary data file that keeps track of all the rest of the files in the
database, in addition to storing data. By convention, the name of a primary data file has the extension MDF.
 Secondary data files A database can have zero or more secondary data files. By convention, the name of a
secondary data file has the extension NDF.
 Log files Every database has at least one log file that contains the information necessary to recover all
transactions in a database. By convention, a log file has the extension LDF.

Each database file has five properties: a logical filename, a physical filename, an initial size, a maximum size, and a
growth increment. The properties of each file, along with other information about the file, are noted in the sysfiles
table (shown in Table 5-1), which contains one row for each file used by a database.

Table 5-1. The sysfiles table.

Column Name Description
fileid The file identification number (unique for each database).
groupid The filegroup identification number.
size The size of the file (in 8-KB pages).
The maximum file size (in 8-KB pages). A value of 0
maxsize indicates no growth, and a value of -1 indicates that the
file should grow until the disk is full.
The growth size of the database. A value of 0 indicates no
growth. Can be either the number of pages or a
growth
percentage of the file size, depending on the value of
status.

0x1 = Default device (unused in SQL Server 2000).

0x2 = Disk file.

0x40 = Log device.

status 0x80 = File has been written to since last backup.

0x4000 = Device created implicitly by CREATE

DATABASE

0x8000 = Device created during database creation.

0x100000 = Growth is in percentage, not pages.

name The logical name of the file.
This document is created with the unregistered version of CHM2PDF Pilot

The name of the physical device, including the full path of

filename
the file.
This document is created with the unregistered version of CHM2PDF Pilot

34

Creating a Database
The easiest way to create a database is to use SQL Server Enterprise Manager, which provides a graphical front end
to Transact-SQL commands and stored procedures that actually create the database and set its properties. Figure
5-1 shows the SQL Server Enterprise Manager Database Properties dialog box, which represents the Transact-SQL
CREATE DATABASE command for creating a new user database. Only someone with the sysadmin role or a user
who's been granted CREATE DATABASE permission by someone with the sysadmin role can issue the CREATE
DATABASE command.

When you create a new user database, SQL Server copies the model database, which—as you just learned—is
simply a template database. For example, if you have an object that you would like created in every subsequent user
database, create that object in model first. (You can also use model to set default database options in all
subsequently created databases.) The model database also includes 19 system tables and 2 system views, which
means that every new database also includes these 21 system objects. SQL Server uses these objects for the
definition and maintenance of each database. The two system views are provided for backward compatibility and
have no current functionality. They mimic information that was available in earlier versions of SQL Server.

Figure 5-1. The Database Properties dialog box, which creates a new database called newdb.

The system objects have names starting with sys. If you haven't added any other objects to model, these 21 system
objects will be the entire contents of a newly created database. Every Transact-SQL command or system stored
procedure that creates, alters, or drops a SQL Server object will result in entries being made to system tables.
WARNING
Do not directly modify the system tables. You might render your database unusable by doing so. Direct modification
is prevented by default: a system administrator must take deliberate action via the sp_configure stored procedure to
allow system tables to be modified directly. I'll discuss system tables in more detail in Chapter 6.

A new user database must be 1 MB or greater in size, and the primary data file size must be at least as large as the
This document is created with the unregistered version of CHM2PDF Pilot

primary data file of the model database. Almost all the possible arguments to the CREATE DATABASE command
have default values so that it's possible to create a database using a simple form of CREATE DATABASE, such as
this:
CREATE DATABASE newdb

This command creates the newdb database, with a default size, on two files whose logical names—newdb and
newdb_log—are derived from the name of the database. The corresponding physical files, newdb.mdf and
newdb_log.ldf, are created in the default data directory (as determined at the time SQL Server was installed).
NOTE
If you use Enterprise Manager to create a database called newdb, the default logical and physical names will be
different than if you use the CREATE DATABASE command. Enterprise Manager will give the data file the logical
name of newdb_Data (instead of just newdb), and the physical file will have the name newdb_data.mdf.

The SQL Server login account that created the database is known as the database owner, and has the user name
DBO when using this database. The default size of the data file is the size of the primary data file of the model
database, and the default size of the log file is half a megabyte. Whether the database name, newdb, is case sensitive
depends on the sort order you chose during setup. If you accepted the default, the name is case insensitive. (Note that
the actual command CREATE DATABASE is case insensitive, regardless of the case sensitivity chosen for data.)

Other default property values apply to the new database and its files. For example, if the LOG ON clause is not
specified but data files are specified, a log file is automatically created with a size that is 25 percent of the sum of the
sizes of all data files.

For the files, if the MAXSIZE clause isn't specified, the file will grow until the disk is full. (In other words, the file size
is considered unlimited.) The values supplied for SIZE, MAXSIZE, and FILEGROWTH can be specified in units of
TB, GB, MB (the default), or KB. The FILEGROWTH property can also be specified as a percentage. A value of 0
for FILEGROWTH indicates no growth. If no FILEGROWTH value is specified at all, the default value is 10
percent, and the minimum value is 64 KB.

A CREATE DATABASE Example

The following is a complete example of the CREATE DATABASE command, specifying three files and all the
properties of each file:
CREATE DATABASE Archive
ON
PRIMARY
( NAME = Arch1,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\archdat1.mdf',
SIZE = 100MB,
MAXSIZE = 200,
FILEGROWTH = 20),
( NAME = Arch2,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\archdat2.ndf',
SIZE = 100MB,
MAXSIZE = 200,
FILEGROWTH = 20)
LOG ON
( NAME = Archlog1,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\archlog1.ldf',
SIZE = 100MB,
MAXSIZE = 200,
This document is created with the unregistered version of CHM2PDF Pilot

FILEGROWTH = 20)
This document is created with the unregistered version of CHM2PDF Pilot

34

Expanding and Shrinking a

Database
Databases can be expanded and shrunk either automatically or manually. The mechanism for automatic expansion is
completely different from the mechanism for automatic shrinkage. Manual expansion is also handled differently than
manual shrinkage. Log files have their own rules for growing and shrinking, so I'll discuss changes in log file size in a
separate section.

Automatic File Expansion

Expansion of a database can happen automatically to any one of the database's files when that particular file becomes
full. The file property FILEGROWTH determines how that automatic expansion happens. The FILEGROWTH
specified when the file is first defined can be qualified using the suffix MB, KB, or % and is always rounded up to the
nearest 64 KB. If the value is specified as a percent, the growth increment is the specified percentage of the size of
the file when the expansion occurs. The file property MAXSIZE sets an upper limit on the size.

Manual File Expansion

Manual expansion of a database file is accomplished using the ALTER DATABASE command to change the SIZE
property of one or more of the files. When you alter a database, the new size of a file must be larger than the current
size. To decrease the size of files, you use the DBCC SHRINKFILE command, which I'll tell you about shortly.

Automatic File Shrinkage

The database property autoshrink allows a database to shrink automatically. The effect is the same as doing a
DBCC SHRINKDATABASE (dbname, 25). This option leaves 25 percent free space in a database after the shrink,
and any free space beyond that is returned to the operating system. The thread that performs autoshrink—which
always has server process ID (spid) 6—shrinks databases at 30-minute intervals. I'll discuss the DBCC
SHRINKDATABASE command in more detail momentarily.

Manual File Shrinkage

You can manually shrink a database using the following two DBCC commands:

DBCC SHRINKFILE ( {file_name | file_id }

[, target_size][, {EMPTYFILE | NOTRUNCATE | TRUNCATEONLY} ] )

DBCC SHRINKDATABASE (database_name [, target_percent]

[, {NOTRUNCATE | TRUNCATEONLY} ] )

DBCC SHRINKFILE
DBCC SHRINKFILE allows you to shrink files in the current database. When target_size is specified, DBCC
SHRINKFILE attempts to shrink the specified file to the specified size in megabytes. Used pages in the part of the file
This document is created with the unregistered version of CHM2PDF Pilot

to be freed are relocated to available free space in the part of the file retained. For example, for a 15-MB data file, a
DBCC SHRINKFILE with a target_size of 12 causes all used pages in the last 3 MB of the file to be reallocated
into any free slots in the first 12 MB of the file. DBCC SHRINKFILE doesn't shrink a file past the size needed to
store the data. For example, if 70 percent of the pages in a 10-MB data file are used, a DBCC SHRINKFILE
statement with a target_size of 5 shrinks the file to only 7 MB, not 5 MB.

DBCC SHRINKDATABASE
The DBCC SHRINKDATABASE command shrinks all files in a database. The database can't be made smaller than
the model database. In addition, the DBCC SHRINKDATABASE command does not allow any file to be shrunk to
a size smaller than its minimum size. The minimum size of a database file is the initial size of the file (specified when the
database was created) or the size to which the file has been explicitly extended or reduced, using either the ALTER
DATABASE or DBCC SHRINKFILE command. If you need to shrink a database smaller than this minimum size,
you should use the DBCC SHRINKFILE command to shrink individual database files to a specific size. The size to
which a file is shrunk becomes the new minimum size.

The numeric target_percent argument passed to the DBCC SHRINKDATABASE command is a percentage of
free space to leave in each file of the database. For example, if you've used 60 MB of a 100-MB database file, you
can specify a shrink percentage of 25 percent. SQL Server will then shrink the file to a size of 80 MB, and you'll have
20 MB of free space in addition to the original 60 MB of data. In other words, the 80-MB file will have 25 percent of
its space free. If, on the other hand, you've used 80 MB or more of a 100-MB database file, there is no way that
SQL Server can shrink this file to leave 25 percent free space. In that case, the file size remains unchanged.

Because DBCC SHRINKDATABASE shrinks the database on a file-by-file basis, the mechanism used to perform
the actual shrinking is the same as that used with DBCC SHRINKFILE. SQL Server first moves pages to the front of
files to free up space at the end, and then it releases the appropriate number of freed pages to the operating system.
Two options for the DBCC SHRINKDATABASE and DBCC SHRINKFILE commands can force SQL Server to
do either of the two steps just mentioned, while a third option is available only to DBCC SHRINKFILE:
 The NOTRUNCATE option causes all the freed file space to be retained in the database files. SQL Server
only compacts the data by moving it to the front of the file. The default is to release the freed file space to the
operating system.
 The TRUNCATEONLY option causes any unused space in the data files to be released to the operating
system. No attempt is made to relocate rows to unallocated pages. When TRUNCATEONLY is used,
target_size and target_percent are ignored.
 The EMPTYFILE option, available only with DBCC SHRINKFILE, empties the contents of a data file and
moves them to other files in the filegroup.
NOTE
DBCC SHRINKFILE specifies a target size in megabytes. DBCC SHRINKDATABASE specifies a target
percentage of free space to leave in the database.

Both the DBCC SHRINKFILE and DBCC SHRINKDATABASE commands give a report for each file that can be
shrunk. For example, if my pubs database currently has an 8-MB data file and a log file of about the same size, I get
the following report when I issue this DBCC SHRINKDATABASE command:
DBCC SHRINKDATABASE(pubs, 10)
RESULTS:
DbId FileId CurrentSize MinimumSize UsedPages EstimatedPages
------ ------ ----------- ----------- ----------- --------------
5 1 256 80 152 152
5 2 1152 63 1152 56

The current size is the size in pages after any shrinking takes place. In this case, the database file (FileId = 1) was
able to be shrunk to 256 pages of 8 KB each, which is 2 MB. But only 152 pages are used. There could be several
reasons for the difference between used pages and current pages:
This document is created with the unregistered version of CHM2PDF Pilot

 If I asked to leave a certain percentage free, the current size will be bigger than the used pages because of
that free space.
 If the minimum size to which I can shrink a file is bigger than the used pages, the current size cannot become
smaller than the minimum size. (The minimum size of a file is the smaller of the initial creation size and the size
the file has been increased to using the ALTER DATABASE command.)
 If the size of the data file for the model database is bigger than the used pages, the current size cannot
become smaller than the size of model's data file.

For the log file (FileId = 2), the only values that really matter are the current size and the minimum size. The other two
values are basically meaningless for log files because the current size is always the same as the used pages, and
because there is really no simple way to estimate how small a log file can be shrunk down.
This document is created with the unregistered version of CHM2PDF Pilot

34

Changes in Log Size

No matter how many physical files have been defined for the transaction log, SQL Server always treats the log as
one contiguous file. For example, when the DBCC SHRINKDATABASE command determines how much the log
can be shrunk, it does not consider the log files separately but determines the shrinkable size based on the entire log.

The transaction log for any database is managed as a set of virtual log files (VLFs) whose size is determined internally
by SQL Server based on the total size of all the log files and the growth increment used when enlarging the log. A log
always grows in units of entire VLFs and can be shrunk only to a VLF boundary. (Figure 5-2 illustrates a physical log
file along with several VLFs.)

A VLF can exist in one of the following three states:

 Active The active portion of the log begins at the minimum log sequence number (LSN) representing an
active (uncommitted) transaction. The active portion of the log ends at the last LSN written. Any VLFs that
contain any part of the active log are considered active VLFs. (Unused space in the physical log is not part of
any VLF.)
 Recoverable The portion of the log preceding the oldest active transaction is needed only to maintain a
sequence of log backups for restoring the database to a former state.
 Reusable If transaction log backups are not being maintained or if you have already backed up the log,
VLFs prior to the oldest active transaction are not needed and can be reused.

Figure 5-2. The VLFs that make up a physical log file.

SQL Server will assume you're not maintaining a sequence of log backups if any of the following are true:
 You have truncated the log using BACKUP LOG WITH NO_LOG or BACKUP LOG WITH
TRUNCATE_ONLY.
 You have set the database to truncate the log automatically on a regular basis with the database option trunc.
log on chkpt. or by setting the recovery mode to SIMPLE.
 You have never taken a full database backup.

In any of these situations, when SQL Server reaches the end of the physical log file, it starts reusing that space in the
physical file by circling back to the file's beginning. In effect, SQL Server recycles the space in the log file that is no
longer needed for recovery or backup purposes. If a log backup sequence is being maintained, the part of the log
before the minimum LSN cannot be overwritten until those log records have actually been backed up. After the log
backup, SQL Server can circle back to the beginning of the file. Once it has circled back to start writing log records
earlier in the log file, the reusable portion of the log is then between the end of the logical log and the active portion of
the log. Figure 5-3 depicts this cycle.
This document is created with the unregistered version of CHM2PDF Pilot

You can actually observe this behavior in one of the sample databases, such as pubs, as long as you have never
made a full backup of the database. If you have never made any modifications to pubs, the size of its transaction log
file will be just about 0.75 MB. The script below creates a new table in the pubs database, inserts three records, and
then updates those records 1000 times. Each update is an individual transaction, and each one is written to the
transaction log. However, you should note that the log does not grow at all, even after 3000 update records are
written. (If you've already taken a backup of pubs, you might want to re-create the database before trying this
example. You can do that by running the script instpubs.sql in the folder \Program Files\Microsoft SQL
Server\MSSQL\install. If your SQL Server is a named instance, you'll need to replace MSSQL with the string
MSSQL$<instance_name>.)

Figure 5-3. The active portion of the log circling back to the beginning of the physical log file.
CREATE TABLE newtable (a int)
GO
INSERT INTO newtable VALUES (10)
INSERT INTO newtable VALUES (20)
INSERT INTO newtable VALUES (30)
GO
DECLARE @counter int
SET @counter = 1
WHILE @counter < 1000 BEGIN
UPDATE newtable SET a = a + 1
SET @counter = @counter + 1
END

Now make a backup of the pubs database after making sure that the database is not in the SIMPLE recovery mode.
I'll discuss recovery modes later in this chapter, but for now, you can just make sure that pubs is in the appropriate
recovery mode by executing the following command:
ALTER DATABASE pubs SET RECOVERY FULL

You can use the following statement to make the backup, substituting the path shown with the path to your SQL
Server installation:
BACKUP DATABASE pubs to disk =
'c:\Program Files\Microsoft SQL Server\MSSQL\backup\pubs.bak'

Run the update script again, starting with the DECLARE statement. You should see that the physical log file has
grown to accommodate the log records added. The initial space in the log could not be reused because SQL Server
assumed that you were saving that information for backups.

Now you can try to shrink the log back down again. If you issue the following command or if you issue the DBCC
This document is created with the unregistered version of CHM2PDF Pilot

SHRINKFILE command for the log file, SQL Server will mark a shrinkpoint in the log but no actual shrinking will
take place until log records are freed by either backing up or truncating the log.
DBCC SHRINKDATABASE (pubs)

You can truncate the log with this statement:

BACKUP LOG pubs WITH TRUNCATE_ONLY

At this point, you should notice that the physical size of the log file has been reduced. If a log is truncated without any
prior shrink command issued, SQL Server marks the space used by the truncated records as available for reuse but it
does not change the size of the physical file.

In the previous version of SQL Server, running the preceding commands exactly as specified did not always shrink
the physical log file. The cases in which the log file did not shrink happened when the active part of the log was
located at the end of the physical file. Physical shrinking can take place only from the end of the log, and the active
portion is never shrinkable. To remedy this situation, you had to enter some "dummy" transactions after truncating the
log, to force the active part of the log to move around to the beginning of the file. In SQL Server 2000, this process is
unnecessary. If a shrink command has already been issued, truncating the log internally generates a series of NO-OP
log records that force the active log to move from the physical end of the file. Shrinking happens as soon as the log is
no longer needed.

Log Truncation
If a database log backup sequence is not being maintained for a database, you can set the database into log truncate
mode by setting the recovery mode to SIMPLE or by using the older mechanism of setting the database option trunc.
log on chkpt. to TRUE. The log is thus truncated every time it gets "full enough." (I'll explain this in a moment.)

Truncation means that all log records prior to the oldest active transaction are removed. It does not necessarily imply
shrinking of the physical log file. In addition, if your database is a publisher in a replication scenario, the oldest open
transaction could be a transaction marked for replication that has not yet been replicated.

"Full enough" means that there are more log records than can be redone during system startup in a reasonable amount
of time, which is referred to as the "recovery interval." You can manually change the recovery interval by using the
sp_configure stored procedure or by using SQL Server Enterprise Manager (right-click the server, select Properties,
click the Database Settings tab, and set the value in the Recovery Interval box), but it is recommend that you let SQL
Server autotune this value. In most cases, this recovery interval value is set to 1 minute. (SQL Server Enterprise
Manager shows zero minutes by default, meaning SQL Server will autotune the value.) SQL Server bases its
recovery interval on the estimation that 10 MB worth of transactions can be recovered in 1 minute.

The actual log truncation is invoked by the checkpoint process, which is usually sleeping and is only woken up on
demand. Each time a user thread calls the log manager, the log manager checks the size of the log. If the size exceeds
the amount of work that can be recovered during the recovery interval, the checkpoint thread is woken up. The
checkpoint thread checkpoints the database and then truncates the inactive portion.

In addition, if the log ever gets to 70 percent full, the log manager wakes up the checkpoint thread to force a
checkpoint. Growing the log is much more expensive than truncating it, so SQL Server truncates the log whenever it
can.
NOTE
If the log is in auto truncate mode and the autoshrink option is set, the log will be physically shrunk at regular
intervals.
This document is created with the unregistered version of CHM2PDF Pilot

If a database has the autoshrink option on, an autoshrink process kicks in every 30 minutes and determines the size to
which the log should be shrunk. The log manager accumulates statistics on the maximum amount of log space used in
the 30-minute interval between shrinks. The autoshrink process marks the shrinkpoint of the log as 125 percent of the
maximum log space used or the minimum size of the log, whichever is larger. (Minimum size is the creation size of the
log or the size to which it has been manually increased or decreased.) The log then shrinks to that size whenever it
gets the chance, which is when the log gets truncated or backed up. It's possible to have autoshrink without having the
database in auto truncate mode, although there's no way to guarantee that the log will actually shrink. For example, if
the log is never backed up, it will never be cleared.
This document is created with the unregistered version of CHM2PDF Pilot

34

Using Database Filegroups

You can group data files for a database into filegroups for allocation and administration purposes. In some cases, you
can improve performance by controlling the placement of data and indexes into specific filegroups on specific disk
drives. The filegroup containing the primary data file is called the primary filegroup. There is only one primary
filegroup, and if you don't specifically ask to place files in other filegroups when you create your database, all your
data files will be in the primary filegroup.

In addition to the primary filegroup, a database can have one or more user-defined filegroups. You can create
user-defined filegroups by using the FILEGROUP keyword in the CREATE DATABASE or ALTER DATABASE
statement.

Don't confuse the primary filegroup and the primary file:

 The primary file is always the first file listed when you create a database, and it typically has the file extension
MDF. The one special feature of the primary file is that its header contains information about all the other files
in the database.
 The primary filegroup is always the filegroup that contains the primary file. This filegroup contains the primary
data file and any files not put into another specific filegroup. All pages from system tables are always allocated
from files in the primary filegroup.

The Default Filegroup

One filegroup always has the property of DEFAULT. Note that DEFAULT is a property of a filegroup and not a
name. Only one filegroup in each database can be the default filegroup. By default, the primary filegroup is the also the
default filegroup. A database owner can change which filegroup is the default by using the ALTER DATABASE
statement. The default filegroup contains the pages for all tables and indexes that aren't placed in a specific filegroup.

Most SQL Server databases have a single data file in one (default) filegroup. In fact, most users will probably never
know enough about how SQL Server works to know what a filegroup is. As a user acquires greater database
sophistication, she might decide to use multiple devices to spread out the I/O for a database. The easiest way to
accomplish this is to create a database file on a RAID device. Still, there would be no need to use filegroups. At the
next level up the sophistication and complexity scale, the user might decide that she really wants multiple
files—perhaps to create a database that uses more space than is available on a single drive. In this case, she still
doesn't need filegroups—she can accomplish her goals using CREATE DATABASE with a list of files on separate
drives.

More sophisticated database administrators (DBAs) might decide that they want to have different tables assigned to
different drives. Only then will they need to use filegroups. The easiest way to accomplish this goal is to use SQL
Server Enterprise Manager to create the database. SQL Server Enterprise Manager will create the necessary
filegroups, and the user still doesn't have to learn anything about filegroup syntax. Only the most sophisticated users
who want to write scripts that set up databases with multiple filegroups will need to know the underlying details.

Why Use Multiple Files?

You might wonder what the reason would be for creating a database on multiple files located on one physical drive.
There's no performance benefit in doing so, but it gives you added flexibility in two important ways.

First, if you need to restore a database from a backup because of a disk crash, the new database must contain the
same number of files as the original. For example, if your original database consisted of one large 12-GB file, you
This document is created with the unregistered version of CHM2PDF Pilot

would need to restore it to a database with one file of that size. If you don't have another 12-GB drive immediately
available, you cannot restore the database! If, however, you originally created the database on several smaller files,
you have added flexibility during a restoration. You might be more likely to have several 4-GB drives available than
one large 12-GB drive.

Second, spreading the database onto multiple files, even on the same drive, gives you the flexibility of easily moving
the database onto separate drives if you modify your hardware configuration in the future. Microsoft's internal SAP
system uses a SQL Server database created on 12 files. Microsoft has found that this provides the ultimate flexibility.
They could separate the files into two groups of six, six groups of two, four groups of three, and so on, which would
allow them to experiment with performance enhancements gained as files are spread over different numbers of
physical drives.

You can also use filegroups to allow backups of only parts of the database at one time. However, if you create an
index in a filegroup that's different from the filegroup the table resides in, you must back up both filegroups (the
filegroup containing the table and the filegroup containing the index). If you create more than one index in a filegroup
that's different from the filegroups in which the corresponding tables reside, you must immediately back up all
filegroups to accommodate these differing filegroups. The BACKUP statement detects all these filegroup situations
and communicates to the user the minimum filegroups that must be backed up.

When you add space to objects stored in a particular filegroup, the data is stored in a proportional fill manner,
which means that if you have one file in a filegroup with twice as much free space as another, the first file will have two
extents (or units of space) allocated from it for each extent allocated from the second file. I'll discuss extents in more
detail later in this chapter.

A FILEGROUP CREATION Example

This example creates a database named sales with three filegroups:
 The primary filegroup with the files Spri1_dat and Spri2_dat. The FILEGROWTH increment for these files is
specified as 15 percent.
 A filegroup named SalesGroup1 with the files SGrp1Fi1 and SGrp1Fi2.
 A filegroup named SalesGroup2 with the files SGrp2Fi1 and SGrp2Fi2.
CREATE DATABASE Sales
ON PRIMARY
( NAME = SPri1_dat,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\SPri1dat.mdf',
SIZE = 10,
MAXSIZE = 50,
FILEGROWTH = 15% ),
( NAME = SPri2_dat,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\SPri2dat.ndf',
SIZE = 10,
MAXSIZE = 50,
FILEGROWTH = 15% ),
FILEGROUP SalesGroup1
( NAME = SGrp1Fi1_dat,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\SG1Fi1dt.ndf',
SIZE = 10,
MAXSIZE = 50,
FILEGROWTH = 5 ),
( NAME = SGrp1Fi2_dat,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\SG1Fi2dt.ndf',
SIZE = 10,
MAXSIZE = 50,
This document is created with the unregistered version of CHM2PDF Pilot

FILEGROWTH = 5 ),
FILEGROUP SalesGroup2
( NAME = SGrp2Fi1_dat,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\SG2Fi1dt.ndf',
SIZE = 10,
MAXSIZE = 50,
FILEGROWTH = 5 ),
( NAME = SGrp2Fi2_dat,
FILENAME =
'c:\program files\microsoft sql server\mssql\data\SG2Fi2dt.ndf',
SIZE = 10,
MAXSIZE = 50,
FILEGROWTH = 5 )
LOG ON
( NAME = 'Sales_log',
FILENAME =
'c:\program files\microsoft sql server\mssql\data\saleslog.ldf',
SIZE = 5MB,
MAXSIZE = 25MB,
FILEGROWTH = 5MB )
This document is created with the unregistered version of CHM2PDF Pilot

34

Altering a Database
You can use the ALTER DATABASE statement to change a database's definition in one of the following ways:
 Change the name of the database.
 Add one or more new data files to the database, which you can optionally put in a user-defined filegroup.
You must put all files added in a single ALTER DATABASE statement in the same filegroup.
 Add one or more new log files to the database.
 Remove a file or a filegroup from the database. You can do this only if the file or filegroup is completely
empty. Removing a filegroup removes all the files in it.
 Add a new filegroup to a database.
 Modify an existing file in one of the following ways:
 Increase the value of the SIZE property.
 Change the MAXSIZE or FILEGROWTH properties.
 Change the name of a file by specifying a NEWNAME property. The value given for NEWNAME is
then used as the NAME property for all future references to this file.
 Change the FILENAME property for files only in the tempdb database; this change doesn't go into effect
until you stop and restart SQL Server. You can change the FILENAME in order to move the tempdb
files to a new physical location.
 Modify an existing filegroup in one of the following ways:
 Mark the filegroup as READONLY so that updates to objects in the filegroup aren't allowed. The
primary filegroup cannot be made READONLY.
 Mark the filegroup as READWRITE, which reverses the READONLY property.
 Mark the filegroup as the default filegroup for the database.
 Change the name of the filegroup.
 Change one or more database options. (I'll discuss database options later in the chapter.)

The ALTER DATABASE statement can make only one of the changes described each time it is executed. Note that
you cannot move a file from one filegroup to another.

ALTER DATABASE Examples

The following examples demonstrate some of the changes you can make using the statement ALTER DATABASE.

This example increases the size of a database file:

USE master
GO
ALTER DATABASE Test1
MODIFY FILE
( NAME = 'test1dat3',
SIZE = 20MB)

The following example creates a new filegroup in a database, adds two 5-MB files to the filegroup, and makes the
new filegroup the default filegroup. We need three ALTER DATABASE statements.
ALTER DATABASE Test1
ADD FILEGROUP Test1FG1
GO
ALTER DATABASE Test1
ADD FILE
This document is created with the unregistered version of CHM2PDF Pilot

( NAME = 'test1dat3',
FILENAME =
'c:\program files\microsoft sql server\mssql\data\t1dat3.ndf',
SIZE = 5MB,
MAXSIZE = 100MB,
FILEGROWTH = 5MB),
( NAME = 'test1dat4',
FILENAME =
'c:\program files\microsoft sql server\mssql\data\t1dat4.ndf',
SIZE = 5MB,
MAXSIZE = 100MB,
FILEGROWTH = 5MB)
TO FILEGROUP Test1FG1
GO
ALTER DATABASE Test1
MODIFY FILEGROUP Test1FG1 DEFAULT
GO
This document is created with the unregistered version of CHM2PDF Pilot

34

Databases Under the Hood

A database consists of user-defined space for the permanent storage of user objects such as tables and indexes. This
space is allocated in one or more operating system files.

Databases are divided into logical pages (of 8 KB each), and within each file the pages are numbered contiguously
from 0 to x, with the upper value x being defined by the size of the file. You can refer to any page by specifying a
database ID, a file ID, and a page number. When you use the ALTER DATABASE command to enlarge a file, the
new space is added to the end of the file. That is, the first page of the newly allocated space is page x + 1 on the file
you're enlarging. When you shrink a database by using either the DBCC SHRINKDATABASE or DBCC
SHRINKFILE command, pages are removed starting at the highest page in the database (at the end) and moving
toward lower-numbered pages. This ensures that page numbers within a file are always contiguous.

The master database contains 50 system tables: the same 19 tables found in user databases, plus 31 tables that are
found only in master. Many of these tables—including syslockinfo, sysperfinfo, syscurconfigs, and sysprocesses
—don't physically exist in the master database; rather, they're built dynamically each time a user queries them.
Twelve of these tables contain information about remote servers; they don't contain any data until you define other
servers with which you'll be communicating. The master database also contains 26 system views. Twenty of these are
INFORMATION_SCHEMA views; I'll discuss them in Chapter 6. Except for the INFORMATION_SCHEMA
views, all these system objects, in both the master and user databases, have names that begin with sys. To see a list
of the system tables and views that are found in only the master database, you can execute the following query:
SELECT type, name FROM master..sysobjects
WHERE type IN ('s', 'v') AND name NOT IN
(SELECT name FROM model..sysobjects)
GO

The master database also contains nine tables that can be referred to as pseudo-system tables. These table names
begin with spt_ and are used as lookup tables by various system procedures, but they aren't true system tables. You
should never modify them directly because that might break some of the system procedures. However, deleting them
doesn't invalidate the basic integrity of the database, which is what happens if true system tables are altered.

When you create a new database using the CREATE DATABASE statement, it is given a unique database ID, or
dbid, and a new row is inserted in the table master..sysdatabases for that database. Only the master database
contains a sysdatabases table. Figure 5-4 below depicts a sample sysdatabases table, and Table 5-2 shows its
columns.

The rows in sysdatabases are updated when a database's ownership or name is changed or when database options
are changed. (I'll discuss this in more detail later in the chapter.)

Table 5-2. Columns of the sysdatabases table.

Column Information
name Name of the database.
Unique database ID; can be reused when the database is
dbid
dropped.
sid System ID of the database creator.
Locking mode; used internally while a database is being
mode
created.
This document is created with the unregistered version of CHM2PDF Pilot

Bit mask that shows whether a database is read-only, off

line, designated for use by a single user only, and so on.
Some of the bits can be set by a database owner using the
status
ALTER DATABASE command; others are set internally.
(The SQL Server documentation shows most of the
possible bit-mask values.)
Another bit mask-like status, with bits indicating additional
status2
database options.
For user databases, the date when the database was
created. For tempdb, this is the date and time that SQL
Server was last started. For other system databases, this
crdate date is not really useful. Depending on decisions made
during installation, it could be the date Microsoft originally
created the database prior to shipping the code, or it
could be the date that you installed SQL Server.
reserved Reserved for future use.
Another bit mask-like status. Contains information about
category
whether the database is involved with replication.
Compatibility level for the database. (I'll discuss this
cmptlevel
concept briefly at the end of the chapter.)
filename Operating system path and name of the primary file.
Internal version of SQL Server that was used to create
version
the database.

Figure 5-4. A partial listing of a sysdatabases table.

Space Allocation
The space in a database is used for storing tables and indexes. The space is managed in units called extents. An
extent is made up of eight logically contiguous pages (or 64 KB of space). To make space allocation more efficient,
SQL Server 2000 doesn't allocate entire extents to tables with small amounts of data. SQL Server 2000 has two
types of extents:
 Uniform extents These are owned by a single object; all eight pages in the extent can be used only by the
owning object.
 Mixed extents These are shared by up to eight objects.

SQL Server allocates pages for a new table or index from mixed extents. When the table or index grows to eight
pages, all future allocations use uniform extents.

When a table or index needs more space, SQL Server needs to find space that's available to be allocated. If the
table or index is still less than eight pages total, SQL Server must find a mixed extent with space available. If the table
or index is eight pages or larger, SQL Server must find a free uniform extent.

SQL Server uses two special types of pages to record which extents have been allocated and which type of use
(mixed or uniform) the extent is available for:
 Global Allocation Map (GAM) pages These pages record which extents have been allocated for any type
This document is created with the unregistered version of CHM2PDF Pilot

of use. A GAM has a bit for each extent in the interval it covers. If the bit is 0, the corresponding extent is in
use; if the bit is 1, the extent is free. Since there are almost 8000 bytes, or 64,000 bits, available on the page
after the header and other overhead are accounted for, each GAM can cover about 64,000 extents, or
almost 4 GB of data. This means that one GAM page exists in a file for every 4 GB of size.
 Shared Global Allocation Map (SGAM) pages These pages record which extents are currently used as
mixed extents and have at least one unused page. Just like a GAM, each SGAM covers about 64,000
extents, or almost 4 GB of data. The SGAM has a bit for each extent in the interval it covers. If the bit is 1,
the extent is being used as a mixed extent and has free pages; if the bit is 0, the extent isn't being used as a
mixed extent, or it's a mixed extent whose pages are all in use.

Table 5-3 shows the bit patterns that each extent has set in the GAM and SGAM based on its current use.

Table 5-3. Bit settings in GAM and SGAM pages.

Current Use of Extent GAM Bit Setting SGAM Bit Setting
Free, not in use 1 0
Uniform extent, or full mixed extent 0 0
Mixed extent with free pages 0 1

If SQL Server needs to find a new, completely unused extent, it can use any extent with a corresponding bit value of
1 in the GAM page. If it needs to find a mixed extent with available space (one or more free pages), it finds an extent
with a value in the GAM of 0 and a value in the SGAM of 1.

SQL Server can quickly locate the GAMs in a file because a GAM is always the third page in any database file
(page 2). An SGAM is the fourth page (page 3). Another GAM appears every 511,230 pages after the first GAM on
page 2, and another SGAM every 511,230 pages after the first SGAM on page 3. Page 0 in any file is the File
Header page, and only one exists per file. Page 1 is a Page Free Space (PFS) page (which I'll discuss shortly). In
Chapter 6, I'll say more about how individual pages within a table look. For now, because I'm talking about space
allocation, I'll examine how to keep track of which pages belong to which tables.

Index Allocation Map (IAM) pages map the extents in a database file used by a heap or index. Recall from Chapter
3 that a heap is a table without a clustered index. Each heap or index has one or more IAM pages that record all the
extents allocated to the object. A heap or index has at least one IAM for each file on which it has extents. A heap or
index can have more than one IAM on a file if the range of the extents exceeds the range that an IAM can record.

An IAM contains a small header, eight page-pointer slots, and a set of bits that map a range of extents onto a file.
The header has the address of the first extent in the range mapped by the IAM. The eight page-pointer slots might
contain pointers to pages belonging to the relevant objects that are contained in mixed extents; only the first IAM for
an object has values in these pointers. Once an object takes up more than eight pages, all its extents are uniform
extents—which means that an object will never need more than eight pointers to pages in mixed extents. If rows have
been deleted from a table, the table can actually use fewer than eight of these pointers. Each bit of the bitmap
represents an extent in the range, regardless of whether the extent is allocated to the object owning the IAM. If a bit is
on, the relative extent in the range is allocated to the object owning the IAM; if a bit is off, the relative extent isn't
allocated to the object owning the IAM.

For example, if the bit pattern in the first byte of the IAM is 1100 0000, the first and second extents in the range
covered by the IAM are allocated to the object owning the IAM and extents 3 through 8 aren't allocated to the
object owning the IAM.

IAM pages are allocated as needed for each object and are located randomly in the database file. Each IAM covers
a possible range of about 512,000 pages. Sysindexes.FirstIAM points to the first IAM page for an object. All the
IAM pages for that object are linked in a chain.
NOTE
In a heap, the data pages and the rows within them aren't in any specific order and aren't linked together. The only
This document is created with the unregistered version of CHM2PDF Pilot

logical connection between data pages is recorded in the IAM pages.

Once extents have been allocated to an object, SQL Server can use pages in those extents to insert new data. If the
data is to be inserted into a B-tree, the location of the new data is based on the ordering within the B-tree. If the data
is to be inserted into a heap, the data can be inserted into any available space. PFS pages within a file record whether
an individual page has been allocated and the amount of space free on each page. Each PFS page covers 8088
contiguous pages (almost 64 MB). For each page, the PFS has 1 byte recording whether the page is empty, 1-50
percent full, 51-80 percent full, 81-95 percent full, or 96-100 percent full. SQL Server uses this information when it
needs to find a page with free space available to hold a newly inserted row. The second page (page 1) of a file is a
PFS page, as is every 8088th page thereafter.

There are also two other kinds of special pages within a data file. The seventh page (page 6) is called a DCM
(Differential Changed Map) page and keeps track of which extents in a file have been modified since the last full
database backup. The eighth page (page 7) of a file is called a BCM (Bulk Changed Map) page and is used when an
extent in the file is used in a minimally or bulk-logged operation. I'll tell you more about these two kinds of pages in the
"Backing Up and Restoring a Database" section later in this chapter. Like GAM and SGAM pages, DCM and BCM
pages have one bit for each extent in the section of the file they represent. They occur at regular intervals, every
511,230 pages.
This document is created with the unregistered version of CHM2PDF Pilot

34

Setting Database Options

Twenty options can be set for a database to control certain behavior within that database. Most of the options must
be set to ON or OFF. By default, all the options that allow only these two values have an initial value of OFF unless
they were set to ON in the model database. All databases created after an option is changed in model will have the
same values as model. You can easily change the value of some of these options by using SQL Server Enterprise
Manager. You can set all of them directly by using the ALTER DATABASE command. You can also use the
sp_dboption system stored procedure to set some, but that procedure is provided for backward compatibility only.
All the options correspond to bits in the status and status2 columns of sysdatabases, although those bits can also
show states that the database owner can't set directly (such as when the database is in the process of being
recovered).

Executing the sp_helpdb stored procedure for a database shows you all the values for the non-Boolean options. For
the Boolean options, the procedure lists the options that are ON. The option values are all listed in the status column
of the output. In addition, the status column of the sp_helpdb output provides the database collation and sort order.
The procedure also returns other useful information, such as database size, creation date, and database owner.
Executing sp_helpdb with no parameters shows information about all the databases in that installation. The following
databases exist on a new default installation of SQL Server, and sp_helpdb produces this output (although the
created dates and sizes can vary):
> EXEC sp_helpdb
name db_size owner dbid created status
--------- -------- ----- ---- ----------- ---------------------------------
master 11.94 MB sa 1 Jul 31 2000 Status=ONLINE, Updateability=R...
model 1.13 MB sa 3 Jul 31 2000 Status=ONLINE, Updateability=R...
msdb 13.00 MB sa 4 Jul 31 2000 Status=ONLINE, Updateability=R...
Northwind 3.63 MB sa 6 Jul 31 2000 Status=ONLINE, Updateability=R...
pubs 2.00 MB sa 5 Jul 31 2000 Status=ONLINE, Updateability=R...
tempdb 8.50 MB sa 2 Aug 21 2000 Status=ONLINE, Updateability=R...

The database options are divided into five categories: state options, cursor options, auto option, SQL options, and
recovery options. Some of the options, in particular the SQL options, have corresponding SET options that you can
turn on or off for a particular connection. Be aware that the ODBC or OLE DB drivers turn a number of these SET
options on by default, so applications will act as though the corresponding database option has already been set. (
Chapter 6 goes into more detail about the SET options that are set by the ODBC and OLE DB drivers and the
interaction with the database options.)

Here is a list of all 20 options, by category. Options listed on a single line are mutually exclusive.
 State options

SINGLE_USER | RESTRICTED_USER | MULTI_USER

OFFLINE | ONLINE
READ_ONLY | READ_WRITE
 Cursor options

CURSOR_CLOSE_ON_COMMIT { ON | OFF }
CURSOR_DEFAULT { LOCAL | GLOBAL }
 Auto options

AUTO_CLOSE {ON | OFF }

This document is created with the unregistered version of CHM2PDF Pilot

AUTO_CREATE_STATISTICS {ON | OFF }

AUTO_SHRINK {ON | OFF }
AUTO_UPDATE_STATISTICS {ON | OFF }
 SQL options

ANSI_NULL_DEFAULT { ON | OFF }
ANSI_NULLS { ON | OFF }
ANSI_PADDING { ON | OFF }
ANSI_WARNINGS { ON | OFF }
ARITHABORT { ON | OFF }
CONCAT_NULL_YIELDS_NULL { ON | OFF }
NUMERIC_ROUNDABORT { ON | OFF }
QUOTED_IDENTIFIER { ON | OFF }
RECURSIVE_TRIGGERS { ON | OFF }
 Recovery options

RECOVERY { FULL | BULK_LOGGED | SIMPLE }

TORN_PAGE_DETECTION { ON | OFF }

The following sections describe all the options except RECOVERY, which I'll discuss in detail later, in the section
titled "Backing Up and Restoring a Database."

State Options
The state options control the usability of the database, in terms of who can use the database and for what operations.
There are three aspects to usability: the user access state option determines which users can use the database, the
status state option determines whether the database is available to anybody for use, and the updateability state option
determines what operations can be performed on the database. You control each of these aspects by using the
ALTER DATABASE command to enable an option for the database. None of the state options uses the keywords
ON and OFF to control the state value.

SINGLE_USER | RESTRICTED_USER | MULTI_USER

These three options describe the user access property of a database. They are mutually exclusive; setting any one of
them unsets the others. To set one of these options for your database, you just use the option name. For example, to
set the pubs database to single user mode, you use the following code:
ALTER DATABASE pubs SINGLE_USER

A database in SINGLE_USER mode can have only one connection at a time. A database in RESTRICTED_USER
mode can have connections only from users who are considered "qualified." A qualified user is any member of the
dbcreator or sysadmin server roles or any member of the db_owner role for that database. The default for a database
is MULTI_USER mode, which means anyone with a valid user name in the database can connect to it. If you attempt
to change a database's state to a mode that is incompatible with the current conditions—for example, if you try to
change the database to SINGLE_USER mode when other connections exist—SQL Server's behavior is determined
by the TERMINATION option you specify. I'll discuss termination options shortly.

To determine which user access value is set for a database, you can use the DATABASEPROPERTYEX() function,
as shown here:
SELECT DATABASEPROPERTYEX('<name of database>', 'UserAccess')
This document is created with the unregistered version of CHM2PDF Pilot

In previous versions of SQL Server, database access was controlled using the procedure sp_dboption and setting
the value of the options dbo use only or single user. If both of these options had a value of false, the database was in
MULTI_USER mode. If the option single user was set to true, the database was in SINGLE_USER mode, and if
the option dbo use only was set to true and single user was set to false, the database was in a state similar to
RESTRICTED_USER. In SQL Server 2000, the user access value is still determined by the same bits in the s
ysdatabases.status field that were used in previous versions to determine single user or dbo use only mode. If the
12 bit is set (with a value of 4096), the database is in SINGLE_USER mode, and if the 11 bit is set (with a value of
2048), the database is in RESTRICTED_USER mode. In SQL Server 2000, you cannot have both of these bits set
at the same time.

OFFLINE | ONLINE
These two options are the status options; the status property of a database is described using one of these option
names. They are mutually exclusive. The default is for a database to be ONLINE. As with the user access options,
when you use ALTER DATABASE to put the database in one of these modes, you don't specify a value of ON or
OFF—you just use the name of the option. When a database is set to OFFLINE, the database is closed and shut
down cleanly and marked as off line. The database cannot be modified while the database is off line. A database
cannot be put into OFFLINE mode if there are any connections in the database. Whether SQL Server waits for the
other connections to terminate or generates an error message is determined by the TERMINATION option specified.

The following code examples show how to set a database's status value to OFFLINE and how to determine the
status of a database:
ALTER DATABASE pubs SET OFFLINE
SELECT DATABASEPROPERTYEX('pubs', 'status')

DATABASEPROPERTYEX() could return status values other than OFFLINE and ONLINE, but those values are
not directly settable using ALTER DATABASE. A database can have the status value RESTORING while it is in the
process of being restored from a backup. It can have the status value RECOVERING during a restart of SQL
Server. The restore process is done on one database at a time, and until SQL Server has finished restoring a
database, the database has a status of RECOVERING. It can have a status of SUSPECT if the recovery process
could not be completed for some reason—the most common ones being that one or more of the log files for the
database is unavailable or unreadable. Each of the five status values corresponds to a different bit set in the
sysdatabases.status column.

READ_ONLY | READ_WRITE
These two options are the updateability options; they describe the updateability property of a database. They are
mutually exclusive. The default is for a database to be READ_WRITE. As with the user access options, when you
use ALTER DATABASE to put the database in one of these modes, you don't specify a value of ON or OFF, you
just use name of the option. When the database is in READ_WRITE mode, any user with the appropriate
permissions can carry out data modification operations. In READ_ONLY mode, no INSERT, UPDATE, or
DELETE operations can be executed. In addition, because no modifications are done when a database is in
READ_ONLY mode, automatic recovery is not run on this database when SQL Server is restarted and no locks
need to be acquired during any SELECT operations. Shrinking a database in READ_ONLY mode is not possible.

A database cannot be put into READ_ONLY mode if there are any connections to the database. Whether SQL
Server waits for the other connections to terminate or generates an error message is determined by the
TERMINATION option specified.

The following code examples show how to set a database's updatability value to READ_ONLY and how to
determine the updatability of a database:
ALTER DATABASE pubs SET READ_ONLY
This document is created with the unregistered version of CHM2PDF Pilot

SELECT DATABASEPROPERTYEX('pubs', 'updateability')

Termination options
As I just mentioned, several of the state options cannot be set when a database is in use or when it is in use by an
unqualified user. You can specify how SQL Server should handle this situation by indicating a termination option in the
ALTER DATABASE command. You can specify that SQL Server wait for the situation to change, that it generate an
error message, or that it terminate the connections of nonqualified users. The termination option determines SQL
Server's behavior in the following situations:
 When you attempt to change a database to SINGLE_USER and it has more than one current connection
 When you attempt to change a database to RESTRICTED_USER and unqualified users are currently
connected to it
 When you attempt to change a database to OFFLINE and there are current connections to it
 When you attempt to change a database to READ_ONLY and there are current connections to it

SQL Server's default behavior in any of these situations is to wait indefinitely. The following TERMINATION options
change this behavior:

ROLLBACK AFTER integer [SECONDS] This option causes SQL Server to wait for the specified number of
seconds and then break unqualified connections. Incomplete transactions are rolled back. When the transition is to
SINGLE_USER mode, unqualified connections are all connections except the one issuing the ALTER DATABASE
statement. When the transition is to RESTRICTED_USER mode, unqualified connections are connections of users
who are not members of the db_owner fixed database role or the dbcreator and sysadmin fixed server roles.

ROLLBACK IMMEDIATE This option breaks unqualified connections immediately. All incomplete transactions
are rolled back. Unqualified connections are the same as those described earlier.

NO_WAIT This option causes SQL Server to check for connections before attempting to change the database state
and causes the ALTER DATABASE statement to fail if certain connections exist. If the database is being set to
SINGLE_USER mode, the ALTER DATABASE statement fails if any other connections exist. If the transition is to
RESTRICTED_USER mode, the ALTER DATABASE statement fails if any unqualified connections exist.

The following command changes the user access option of the pubs database to SINGLE_USER and generates an
error if any other connections to the pubs database exist:
ALTER DATABASE pubs SET SINGLE_USER WITH NO_WAIT

Cursor Options
All of the cursor options control the behavior of server-side cursors that were defined using one of the following
Transact-SQL commands for defining and manipulating cursors: DECLARE, OPEN, FETCH, CLOSE, and
DEALLOCATE. In Chapter 13, I'll discuss Transact-SQL cursors in detail.

CURSOR_CLOSE_ON_COMMIT {ON | OFF}

When this option is set to ON, any open cursors are automatically closed (in compliance with SQL-92) when a
transaction is committed or rolled back. If OFF (the default) is specified, cursors remain open after a transaction is
committed. Rolling back a transaction closes any cursors except those defined as INSENSITIVE or STATIC.

CURSOR_DEFAULT {LOCAL | GLOBAL}

This document is created with the unregistered version of CHM2PDF Pilot

When this option is set to LOCAL and cursors aren't specified as GLOBAL when created, the scope of any cursor
is local to the batch, stored procedure, or trigger in which it was created. The cursor name is valid only within this
scope. The cursor can be referenced by local cursor variables in the batch, stored procedure, or trigger, or by a
stored procedure output parameter. When this option is set to GLOBAL and cursors aren't specified as LOCAL
when created, the scope of the cursor is global to the connection. The cursor name can be referenced in any stored
procedure or batch executed by the connection.

Auto Options
The auto options affect actions that SQL Server might take automatically. All these options are Boolean options, with
values of either ON or OFF.

AUTO_CLOSE
When this option is set to ON (the default when SQL Server runs on Windows 98), the database is closed and shut
down cleanly when the last user of the database exits, thereby freeing any resources. When a user tries to use the
database again, it automatically reopens. If the database was shut down cleanly, the database isn't initialized
(reopened) until a user tries to use the database the next time SQL Server is restarted. The AUTO_CLOSE option is
handy for personal SQL Server databases because it allows you to manage database files as normal files. You can
move them, copy them to make backups, or even e-mail them to other users. However, you shouldn't use this option
for databases accessed by an application that repeatedly makes and breaks connections to SQL Server. The
overhead of closing and reopening the database between each connection will hurt performance.

AUTO_CREATE_STATISTICS
When this option is set to ON (the default), statistics are automatically created by the SQL Server query optimizer on
columns referenced in a query's WHERE clause. Adding statistics improves query performance because the SQL
Server query optimizer can better determine how to evaluate a query.

AUTO_SHRINK
When this option is set to ON, all of a database's files are candidates for periodic shrinking. Both data files and log
files can be automatically shrunk by SQL Server. The only way to free space in the log files so that they can be shrunk
is to back up the transaction log or set the recovery mode to SIMPLE. The log files shrink at the point that the log is
backed up or truncated.

AUTO_UPDATE_STATISTICS
When this option is set to ON (the default), existing statistics are automatically updated if the data in the tables has
changed. SQL Server keeps a counter of the modifications that have been made to a table and uses it to determine
when statistics are outdated. When this option is set to OFF, existing statistics are not automatically updated. (They
can be updated manually.) I'll discuss statistics in much more detail in Chapter 15.

SQL Options
The SQL options control how various SQL statements are interpreted. All of these are Boolean options. Although
the default for all of these options is OFF for SQL Server itself, many tools, such as the SQL Query Analyzer, and
many programming interfaces, such as ODBC, enable certain session-level options that override the database options
and make it appear as if the ON behavior is the default. I'll discuss the interaction of the SET options with the
database options in Chapter 7.
This document is created with the unregistered version of CHM2PDF Pilot

ANSI_NULL_DEFAULT
When this option is set to ON, columns comply with the ANSI-92 rules for column nullability. That is, if you don't
specifically indicate whether a column in a table allows NULL values, NULLs are allowed. When this option is set to
OFF, newly created columns do not allow NULLs if no nullability constraint is specified.

ANSI_NULLS
When this option is set to ON, any comparisons with a NULL value result in UNKNOWN, as specified by the
ANSI-92 standard. If this option is set to OFF, comparisons of non-Unicode values to NULL result in a value of
TRUE if both values being compared are NULL.

ANSI_PADDING
When this option is set to ON, strings being compared to each other are set to the same length before the
comparison takes place. When this option is OFF, no padding takes place.

ANSI_WARNINGS
When this option is set to ON, errors or warnings are issued when conditions such as division by zero or arithmetic
overflow occur.

ARITHABORT
When this option is set to ON, a query is terminated when an arithmetic overflow or division-by-zero error is
encountered during the execution of a query. When this option is OFF, the query returns NULL as the result of the
operation.

CONCAT_NULL_YIELDS_NULL
When this option is set to ON, concatenating two strings results in a NULL string if either of the strings is NULL.
When this option is set to OFF, a NULL string is treated as an empty (zero-length) string for the purposes of
concatenation.

NUMERIC_ROUNDABORT
When this option is set to ON, an error is generated if an expression will result in loss of precision. When this option
is OFF, the result is simply rounded. The setting of ARITHABORT determines the severity of the error. If
ARITHABORT is OFF, only a warning is issued and the expression returns a NULL. If ARITHABORT is ON, an
error is generated and no result is returned.

QUOTED_IDENTIFIER
When this option is set to ON, identifiers such as table and column names can be delimited by double quotation
marks and literals must then be delimited by single quotation marks. All strings delimited by double quotation marks
are interpreted as object identifiers. Quoted identifiers don't have to follow the Transact-SQL rules for identifiers
when QUOTED_IDENTIFIER is ON. They can be keywords and can include characters not normally allowed in
Transact-SQL identifiers, such as spaces and dashes. You can't use double quotation marks to delimit literal string
expressions; you must use single quotation marks. If a single quotation mark is part of the literal string, it can be
represented by two single quotation marks ("). This option must be set to ON if reserved keywords are used for
object names in the database. When it is OFF, identifiers can't be in quotation marks and must follow all
Transact-SQL rules for identifiers.
This document is created with the unregistered version of CHM2PDF Pilot

RECURSIVE_TRIGGERS
When this option is set to ON, triggers can fire recursively, either directly or indirectly. Indirect recursion occurs
when a trigger fires and performs an action that causes a trigger on another table to fire, thereby causing an update to
occur on the original table, which causes the original trigger to fire again. For example, an application updates table T1
, which causes trigger Trig1 to fire. Trig1 updates table T2, which causes trigger Trig2 to fire. Trig2 in turn updates
table T1, which causes Trig1 to fire again. Direct recursion occurs when a trigger fires and performs an action that
causes the same trigger to fire again. For example, an application updates table T3, which causes trigger Trig3 to
fire. Trig3 updates table T3 again, which causes trigger Trig3 to fire again. When this option is OFF (the default),
triggers can't be fired recursively.

Recovery Options
Recovery options control how much recovery can be done on a SQL Server database. The RECOVERY option
itself also controls how much information is logged and how much of the log is available for backups. I'll cover this
option in more detail in the section titled "Backing Up and Restoring a Database" later in this chapter.

TORN_PAGE_DETECTION
When this option is set to ON (the default), it causes a bit to be flipped for each 512byte sector in a database page
(8 KB) whenever the page is written to disk. It allows SQL Server to detect incomplete I/O operations caused by
power failures or other system outages. If a bit is in the wrong state when the page is later read by SQL Server, the
page was written incorrectly. (A torn page has been detected.) Although SQL Server database pages are 8 KB,
disks perform I/O operations using 512-byte sectors. Therefore, 16 sectors are written per database page. A torn
page can occur if the system crashes (for example, because of power failure) between the time the operating system
writes the first 512-byte sector to disk and the completion of the 8-KB I/O operation. If the first sector of a database
page is successfully written before the crash, it will appear that the database page on disk was updated even though
the operation might not have succeeded. Using battery-backed disk caches can ensure that the data is successfully
written to disk or not written at all. In this case, you can set TORN_PAGE_DETECTION to OFF because it isn't
needed. If a torn page is detected, the database must be restored from backup because it will be physically
inconsistent.
This document is created with the unregistered version of CHM2PDF Pilot

34

Other Database Considerations

Keep these additional points in mind about databases in SQL Server.

Databases vs. Schemas

The ANSI SQL-92 standard includes the notion of a schema—or, more precisely, an SQL-schema, which in many
ways is similar to SQL Server's database concept. Per the ANSI standard, an SQL-schema is a collection of
descriptors, each of which is described in the documentation as "a coded description of an SQL object." Basically, a
schema is a collection of SQL objects, such as tables, views, and constraints. ANSI SQL-schemas are similar to
SQL Server databases.

SQL Server 6.5 introduced support for the ANSI SQL-schema. However, the concept of a database in SQL
Server is longstanding and much richer than that of a schema. SQL Server provides more extensive facilities for
working with a database than for working with a schema. SQL Server includes commands, stored procedures, and
powerful tools such as SQL Server Enterprise Manager that are designed around the fundamental concept of a
database. These tools control backup, restoring, security, enumeration of objects, and configuration; counterparts of
these tools don't exist for schemas. The SQL Server implementation of a schema is essentially a check box feature
that provides conformance with the ANSI standard; it's not the preferred choice. Generally speaking, you should use
databases, not schemas.

Using Removable Media

After you've created a database, you can package it so that you can distribute it via removable media such as CD.
This can be useful for distributing large datasets. For example, you might want to put a detailed sales history database
on a CD and send a copy to each of your branch offices. Typically, such a database is read-only (because CDs are
read-only), although this isn't mandatory.

To create a removable media database, you create the database using the stored procedure sp_create_removable
instead of the CREATE DATABASE statement. When calling the procedure, you must specify three or more files
(one for the system catalog tables, one for the transaction log, and one or more for the user data tables). You must
have a separate file for the system tables because when the removable media database is distributed and installed, the
system tables will be installed to a writable device so that users can be added, permissions can be granted, and so on.
The data itself is likely to remain on the read-only device.

Because removable media devices such as CDs are typically slower than hard drives, you can distribute on
removable media a database that will be moved to a hard disk. If you're using a writable removable device, such as
an optical drive, be sure that the device and controller are both on the Hardware Compatibility List (HCL). (You can
find the HCL at www.microsoft.com/hcl.) I also recommend that you run the hard-disk test discussed in Chapter 4 on
any such device. The failure rates of removable media devices are typically higher than those of standard hard disks.

A database can use multiple CDs or removable media devices. However, all media must be available simultaneously.
For example, if a database uses three CDs, the system must have three CD drives so that all discs can be available
when the database is used.

You can use the sp_certify_removable stored procedure to ensure that a database created with the intention of
being burned onto a CD or other removable media meets certain restrictions. The main restriction is that the login sa
must own the database and all the objects must be owned by the user dbo. There can be no users in the database
This document is created with the unregistered version of CHM2PDF Pilot

other than dbo and guest. You can, however, have roles defined in the database, and permissions can be assigned to
those roles. The stored procedure sp_certify_removable ensures that the database was created properly with the
system tables separate from any user tables.

The first time you use a database sent on removable media, you use the sp_attach_db stored procedure to see the
location of each file. You'll probably want to move the file containing the system tables to a writable disk so that you
can create users, stored procedures, and additional permissions. You can keep the data on the removable media if
you won't be modifying it. You can subsequently set the OFFLINE database option using ALTER DATABASE to
toggle the database's availability. This book's companion CD contains a sample script that creates a database, ensures
that it's appropriate for removable media use, and then installs it on your system. However, a database with no tables
or data is pretty useless, so in the next chapter you'll learn how to create tables.

Detaching and Reattaching a Database

The ability to detach and reattach databases offers much broader benefits than just allowing the creation of
removable databases. You can use the procedures sp_detach_db and sp_attach_db to move a database to a new
physical drive—for example, if you're upgrading your hardware. You can use these stored procedures to make a
copy of the database for testing or development purposes or as an alternative to the backup and restore commands.

Detaching a database requires that no one is using the database. If you find existing connections that you can't
terminate, you can use the ALTER DATABASE command and set the database to SINGLE_USER mode using one
of the termination options that automatically breaks existing connections. Detaching a database ensures that there are
no incomplete transactions in the database and that there are no dirty pages for this database in memory. If these
conditions cannot be met, the detach operation will not succeed. Once the database is detached, the entry for it is
removed from the sysdatabases table in the master database, and from SQL Server's perspective, it's as if you had
dropped the database. The command to detach a database is shown here:
EXEC sp_detach_db <name of database>

NOTE
You can also drop the database with the DROP DATABASE command, but using this command is a little more
severe. SQL Server makes sure that no one is connected to the database before dropping it, but no check of dirty
pages or open transactions is made. Dropping a database also removes the physical files from the operating system,
so unless you have a backup, the database is really gone.

The files for a detached database still exist, but the operating system considers them closed files, so they can be
copied, moved, or even deleted like any other operating system files. A database that has been detached can be
reattached using the stored procedure sp_attach_db. This procedure has a few more options than its detaching
counterpart. If all the files still exist in their original locations, which would be the case if you detached the database
just so you could copy the files to another server, all you need to specify is the location of the primary file. Remember
that the primary file's header information contains the location of all the files belonging to the database. In fact, if some
of the files exist in the original locations and only some of them have moved, you must specify only the moved files'
locations when you execute the sp_attach_db procedure.

Although the documentation says that you should use sp_attach_db only on databases that were previously
detached using sp_detach_db, sometimes following this recommendation isn't necessary. If you shut down the SQL
server, the files will be closed, just as if you had detached the database. However, you will not be guaranteed that all
dirty pages from the database were written to disk before the shutdown. This should not cause a problem when you
attach such a database if the log file is available. The log file will have a record of all completed transactions, and a full
recovery will be done when the database is attached to make sure that the database is consistent. One benefit of using
the sp_detach_db procedure is that SQL Server will know that the database was cleanly shut down, and the log file
does not have to be available to attach the database. SQL will build a new log file for you. This can be a quick way to
shrink a log file that has become much larger than you would like, because the new log file that sp_attach_db creates
for you will be the minimum size—less than 1 MB. Note that this trick for shrinking the log will not work if the
This document is created with the unregistered version of CHM2PDF Pilot

database has more than one log file.

Here is the syntax for the sp_attach_db procedure:

sp_attach_db [ @dbname = ] 'dbname' ,

[ @filename1 = ] 'filename_n'
[ ,...16 ]

Note that all you need to specify is the current filenames, regardless of whether the current names are the same as the
original names. SQL Server will find the specified files and use them when attaching the database. You can even
supply a new database name as you're attaching the files. You are limited to specifying up to 16 files for the database.
Remember that you have to specify the filenames only if they are not in the original location stored in the header of the
primary file. If you have a database for which you must specify more than 16 files, you can use the CREATE
DATABASE command and specify the FOR ATTACH option.

Compatibility Levels
SQL Server 7 included a tremendous amount of new functionality and changed certain behaviors that existed in
earlier versions. SQL Server 2000 has added even more new features. To provide the most complete level of
backward compatibility, Microsoft allows you to set the compatibility level of a database to one of four modes: 80,
70, 65, or 60. All newly created databases in SQL Server 2000 have a compatibility level of 80 unless you change
the level for the model database. A database that has been upgraded (using the Upgrade Wizard) from SQL Server
version 6.0 or 6.5 will have its compatibility level set to the SQL Server version under which the database was last
used (either 60 or 65). If you upgrade a server for SQL Server 7 to SQL Server 2000, all the databases on that
server will have their compatibility level set to 80, although you can force SQL Server 2000 to behave like SQL
Server 7 by setting this level to 70.

All the examples and explanations in this book assume that you're using a database that's in 80 compatibility mode
unless otherwise noted. If you find that your SQL statements behave differently than the ones in the book, you should
first verify that your database is in 80 compatibility mode by executing this procedure:

EXEC sp_dbcmptlevel 'database name'

To change to a different compatibility level, run the procedure using a second argument of one of the three modes:

EXEC sp_dbcmptlevel 'database name', compatibility-mode

NOTE
Not all changes in behavior from older versions of SQL Server can be duplicated by changing the compatibility level.
For the most part, the differences have to do with whether new keywords and new syntax are recognized and have
no effect on how your queries are processed internally. For a complete list of the behavioral differences between the
four modes, see the online documentation for the sp_dbcmptlevel procedure.

The compatibility-level options merely provide a transition period while you're upgrading a database or an application
to SQL Server 2000. I strongly suggest that you carefully consider your use of this option and make every effort to
change your applications so that compatibility options are no longer needed. Microsoft doesn't guarantee that these
options will continue to work in future versions of SQL Server.
This document is created with the unregistered version of CHM2PDF Pilot

34

Backing Up and Restoring a

Database
As you're probably aware by now, this book is not intended as a how-to book for database administrators. The
Bibliography lists several excellent books that can teach you the mechanics of actually making database backups and
restoring and can suggest best practices for setting up a backup-and-restore plan for your organization. Nevertheless,
I'd like to discuss some important issues relating to backup and restore processes to help you understand why one
backup plan might be better suited to your needs than another.

Types of Backups
No matter how much fault tolerance you have implemented on your database system, it is no replacement for regular
backups. Backups can provide a solution to accidental or malicious data modifications, programming errors, and
natural disasters (if you store backups in a remote location). If you choose to provide the fastest possible speed for
your data files at the cost of fault tolerance, backups provide insurance in case your data files are damaged.

The process of re-creating a database from backups is called restoring. The degree to which you can restore the lost
data depends on the type of backup. There are three main types of backups in SQL Server 2000, and a couple of
additional variations on those types:
 Full backup A full database backup basically copies all the pages from a database onto a backup device,
which can be a local or network disk file, a local tape drive, or even a named pipe.
 Differential backup A differential backup copies only the extents that were changed since the last full
backup was made. The extents are copied onto a specified backup device. SQL Server can quickly tell
which extents need to be backed up by examining the bits on the DCM pages for each data file in the
database. Each time a full backup is made, all the bits are cleared to 0. When any page in an extent is
changed, its corresponding bit in the DCM page is changed to 1.
 Log backup In most cases, a log backup copies all the log records that have been written to the transaction
log since the last full or log backup was made. However, the exact behavior of the BACKUP LOG command
depends on your database's recovery mode setting. I'll discuss recovery modes shortly.
NOTE
For full details on the mechanics of defining backup devices, making backups, or scheduling backups to occur at
regular intervals, consult SQL Server Books Online or one of the SQL Server administration books listed in the
Bibliography.

A full backup can be made while your SQL Server is in use. This is considered a "fuzzy" backup—that is, it is not an
exact image of the state of the database at any particular point in time. The backup threads just copy extents, and if
other processes need to make changes to those extents while the backup is in progress, they can do so.

To maintain consistency for either a full or a differential backup, SQL Server records the current log sequence
number (LSN) at the time the backup starts and then again at the time the backup ends. This allows the backup to
also capture the relevant parts of the log. The relevant part starts with the oldest open transaction at the time of the
first recorded LSN and ends with the second recorded LSN.

As mentioned previously, what gets recorded with a log backup depends on the recovery model you are using. So
before I talk about log backup in detail, I'll tell you about recovery models.
This document is created with the unregistered version of CHM2PDF Pilot

Recovery Models
As I told you in the section on database options, three values can be set for the RECOVERY option: FULL,
BULK_LOGGED, or SIMPLE. The value you choose determines the speed and size of your transaction log
backups as well as the degree to which you are at risk for loss of committed transactions in case of media failure.

FULL recovery model

The FULL recovery model provides the least risk of losing work in the case of a damaged data file. If a database is
in this mode, all operations are fully logged, which means that in addition to logging every row added with the
INSERT operation, removed with the DELETE operation, or changed with the UPDATE operation, SQL Server
also writes to the transaction log in its entirety every row inserted using a bcp or BULK INSERT operation. If you
experience a media failure for a database file and need to recover a database that was in FULL recovery mode, and
you've been making regular transaction log backups preceded by a full database backup, you can restore to any
specified point in time up to the time of the last log backup. In addition, if your log file is available after the failure of a
data file, you can restore up to the last transaction committed before the failure. SQL Server 2000 also supports a
feature called log marks, which allows you to place reference points in the transaction log. If your database is in
FULL recovery mode, you can choose to recover to one of these log marks. I'll talk a bit more about log marks in
Chapter 12.

In FULL recovery mode, SQL Server will also fully log CREATE INDEX operations. In SQL Server 2000, when
you restore from a transaction log backup that includes index creations, the recovery operation is much faster because
the index does not have to be rebuilt—all the index pages have been captured as part of the database backup. In
previous versions, SQL Server logged only the fact that an index had been built, so when you restored from a log
backup, the entire index would have to be built all over again!

So, FULL recovery mode sounds great, right? As always, there's a tradeoff. The biggest tradeoff is that the size of
your transaction log files can be enormous, and therefore it can take substantially longer to make log backups than
with any previous release.

BULK_LOGGED recovery model

The BULK_LOGGED recovery model allows you to completely restore a database in case of media failure and also
gives you the best performance and least log space usage for certain bulk operations. These bulk operations include
BULK INSERT, bcp, CREATE INDEX, SELECT INTO, WRITETEXT, and UPDATETEXT. In FULL recovery
mode, these operations are fully logged, but in BULK_LOGGED recovery mode, they are only minimally logged.

When you execute one of these bulk operations, SQL Server logs only the fact that the operation occurred.
However, the operation is fully recoverable because SQL Server keeps track of what extents were actually modified
by the bulk operation. Every data file in a SQL Server 2000 database now has an additional allocation page called a
BCM page, which is managed much like the GAM and SGAM pages that I discussed earlier in the chapter. Each bit
on a BCM page represents an extent, and if the bit is 1 it means that this extent has been changed by a minimally
logged bulk operation since the last full database backup. A BCM page is located at the 8th page of every data file,
and every 511,230 pages thereafter. All the bits on a BCM page are reset to 0 every time a full database backup or a
log backup occurs.

Because of the ability to minimally log bulk operations, the operations themselves can be carried out much faster than
in FULL recovery mode. There is a little overhead to setting the bits in the appropriate BCM page, but compared to
the cost of logging each individual change to a data or index row, the cost of flipping bits is almost negligible.

If your database is in BULK_LOGGED mode and you have not actually performed any bulk operations, you can
restore your database to any point in time or to a named log mark because the log will contain a full sequential record
of all changes to your database.
This document is created with the unregistered version of CHM2PDF Pilot

The tradeoff comes during the backing up of the log. In addition to copying the contents of the transaction log to the
backup media, SQL Server scans the BCM pages and backs up all the modified extents along with the transaction
log itself. The log file itself stays small, but the backup of the log can be many times larger. So the log backup takes
more time and might take up a lot more space than in FULL recovery mode. The time it takes to restore a log backup
made in BULK_LOGGED recovery mode is similar to the time it takes to restore a log backup made in FULL
recovery mode. The operations don't have to be redone; all the information necessary to recover all data and index
structures is available in the log backup.

SIMPLE recovery model

The SIMPLE recovery model offers the simplest backup-and-restore strategy. Your transaction log is truncated at
regular, frequent intervals. Therefore, only full database backups and differential backups are allowed. You get an
error if you try to back up the log while in SIMPLE recovery mode. Because the log is not needed for backup
purposes, sections of it can be reused as soon as all the transactions it contains are committed or rolled back, and the
transactions are no longer needed for recovery from server or transaction failure.

Converting from SQL Server 7

Microsoft intended these recovery models to replace the select into/bulkcopy and trunc. log on chkpt. database
options. Earlier versions of SQL Server required that the select into/bulkcopy option be set in order to perform a
SELECT INTO or bulk copy operation. The trunc. log on chkpt. option forced your transaction log to be truncated
every time a checkpoint occurred in the database. This option was recommended only for test or development
systems, not for production servers. You can still set these options using the sp_dboption procedure, but not using
the ALTER DATABASE command. However, in SQL Server 2000, changing either of these options using
sp_dboption also changes your recovery mode, and changing your recovery mode changes the value of one or both
of these options, as you'll see below. The recommended method for changing your database recovery mode is to use
the ALTER DATABASE command:
ALTER DATABASE <database_name>
SET RECOVERY [FULL | BULK_LOGGED | SIMPLE]

To see what mode your database is in, you can use the DATABASEPROPERTYEX() property function:
SELECT DATABASEPROPERTYEX('<database_name>', 'recovery')

As I just mentioned, you can change the recovery mode by changing the database options. For example, if your
database is in FULL recovery mode and you change the select into/bulkcopy option to true, your database recovery
mode automatically changes to BULK_LOGGED. Conversely, if you force the database back into FULL mode using
ALTER DATABASE, the value of the select into/bulkcopy option changes automatically. In fact, sysdatabases
doesn't record any special information for the recovery mode. The recovery mode is determined by the status bits for
these two database options. If bit 3 in sysdatabases.status is set, the database has select into/bulkcopy enabled,
and if bit 4 is set, the database has trunc. log on chkpt. enabled. Table 5-4 shows the relationship between the
database options and the new recovery modes.

Table 5-4. The relationship between SQL Server 7 database options and recovery modes.
If trunc. log on chkpt. is: And select into/bulkcopy is: The recovery mode is:
FALSE FALSE FULL
FALSE TRUE BULK_LOGGED
TRUE FALSE SIMPLE
TRUE TRUE SIMPLE
This document is created with the unregistered version of CHM2PDF Pilot

If you're using SQL Server 2000 Standard or Enterprise Edition, the model database starts in FULL recovery mode,
so all your new databases will also be in FULL mode. If you're using SQL Server Personal Edition or the Microsoft
SQL Server Desktop Engine, the model database starts in SIMPLE recovery mode. You can change the mode of
the model database or any other user database by using the ALTER DATABASE command.

The new recovery model offers you two major benefits over previous versions. First, you can always perform a
SELECT INTO operation without having to worry about what options you've set. Prior to SQL Server 2000, you
could only run the SELECT INTO or minimally logged bulk copy operation if you had set the specific database
option to true, and only a database owner could change that option. That sometimes meant a restriction on what
non-DBO developers could accomplish.

Second, you can freely switch between the FULL and BULK_LOGGED modes without worrying about your
backup scripts failing. Prior to SQL Server 2000, once you performed a SELECT INTO or a bulk copy, you could
no longer back up your transaction log. So if you had automatic log backup scripts scheduled to run at regular
intervals, these would break and generate an error. This can no longer happen. You can run SELECT INTO or bulk
copy in any recovery mode, and you can back up the log in either FULL or BULK_LOGGED mode.

In addition, you can easily switch between FULL and BULK_LOGGED modes if you usually operate in FULL
mode but occasionally need to perform a bulk operation quickly. You can change to BULK_LOGGED and pay the
price later when you back up the log; the backup will simply take longer and be larger.

You can't easily switch to and from SIMPLE mode. When you use the ALTER DATABASE command to change
from SIMPLE to FULL or BULK_LOGGED, you must first make a complete database backup in order for the
change in behavior to be complete. Remember that SIMPLE recovery mode is comparable to the database option
that truncates the log at regular intervals. The truncation option isn't recommended for production databases, where
you need maximum transaction recoverability. The only time that SIMPLE mode is really useful is in test and
development situations or for small databases that are primarily read-only. I suggest that you use FULL or
BULK_LOGGED for your production databases and that you switch between those modes whenever you need to.

Choosing a Backup Type

If you're responsible for creating the backup plan for your data, you'll not only need to choose a recovery mode but
also decide what kind of backup to make. I mentioned the three main types: full, differential, and log. In fact, you can
use all three types together. To accomplish any type of full restoration of a database, you must occasionally make a
full database backup. In addition, you can also make differential or log backups. Here are some facts to help you
decide between these last two:

A differential backup:
 Is faster if your environment includes a lot of changes to the same data. It will back up only the most recent
change, whereas a log backup will capture every individual update.
 Captures the entire B-tree structures for new indexes, whereas a log backup captures each individual step in
building the index.
 Is cumulative. When you recover from a media failure, only the most recent differential backup needs to be
restored because it will contain all the changes since the last full database backup.

A log backup:
 Allows you to restore to any point in time because it is a sequential record of all changes.
 Can be made after a failure of the database media, as long as the log is available. This will allow you to
recover right up to the point of the failure. The last log backup (called the tail of the log) must specify the
WITH NO_TRUNCATE option in the BACKUP LOG command if the database itself is unavailable.
 Is sequential and discrete. Each log backup contains completely different log records. When you use a log
backup to restore a database after a media failure, all log backups must be applied in the order that they were
made.
This document is created with the unregistered version of CHM2PDF Pilot

Restoring a Database
How often you make each type of backup determines two things: how fast you can restore a database and how
much control you have over which transactions are restored. Consider the schedule in Figure 5-5, which shows a
database fully backed up on Sunday. The log is backed up daily, and a differential backup is made on Tuesday and
Thursday. A drive failure occurs on Friday. If the failure does not include the log files or if you have mirrored them
using RAID 1, you should back up the tail of the log with the NO_TRUNCATE option.
WARNING
If you are operating in BULK_LOGGED recovery mode, backing up the log also backs up any data that was
changed with a BULK_LOGGED operation, so you might need to have more than just the log file available to back
up the tail of the log. You'll also need to have available any filegroups containing data inserted with a bulk copy or
SELECT INTO command.

Figure 5-5. Combined usage of log and differential backups reduces total restore time.

To restore this database after a failure, you must start by restoring the full backup made on Sunday. This does two
things: it copies all the data, log, and index pages from the backup media to the database files, and it applies all the
transactions in the log. You must determine whether incomplete transactions are rolled back. You can opt to recover
the database by using the WITH RECOVERY option of the RESTORE command. This will roll back any incomplete
transactions and open the database for use. No further restoring can be done. If you choose not to roll back
incomplete transactions by specifying the WITH NORECOVERY option, the database will be left in an inconsistent
state and will not be usable.

If you choose WITH NORECOVERY, you can then apply the next backup. In the scenario depicted in Figure 5-5,
you would restore the differential backup made on Thursday, which would copy all the changed extents back into the
data files. The differential backup also contains the log records spanning the time the differential backup was being
made, so again you have to decide whether to recover the database. Complete transactions are always rolled
forward, but you determine whether incomplete transactions are rolled back.

After the last differential backup is restored, you must restore, in sequence, all the log backups made after the last
differential backup was made. This includes the tail of the log backed up after the failure if you were able to make this
last backup.
NOTE
The recovery done during a restore operation works almost exactly the same way as restart recovery, which I
described in Chapter 3. There is an analysis pass to determine how much work might need to be done, a roll-forward
pass to redo completed transactions and return the database to the state it was in when the backup was complete,
and a rollback pass to undo incomplete transactions. The big difference between restore recovery and restart
recovery is that with restore recovery you have control over when the rollback pass is done. It should not be done
until all the rolling forward from all the backups has been applied. Only then should you roll back any transactions that
This document is created with the unregistered version of CHM2PDF Pilot

are still not complete.

Backing up and restoring files and filegroups

SQL Server 2000 allows you to back up individual files or filegroups. This can be useful in environments with
extremely large databases. You can choose to back up just one file or filegroup each day, so the entire database does
not have to be backed up as often. It also can be useful when you have an isolated media failure on just a single drive
and you think that restoring the entire database would take too long.

Here are a few details you should keep in mind when backing up and restoring files and filegroups:
 Individual files and filegroups can be backed up only when your database is in FULL or BULK_LOGGED
recovery mode because you must apply log backups after you restore a file or filegroup and you can't make
log backups in SIMPLE mode.
 Unlike differential and full database backups, a backup of a file or filegroup does not back up any portion of
the transaction log.
 You can restore individual file or filegroup backups from a full database backup.
 Immediately before restoring an individual file or filegroup, you must back up the transaction log. You must
have an unbroken chain of log backups from the time the file or filegroup backup was made.
 After restoring a file or filegroup backup, you must restore all the transaction logs made between the time you
backed up the file or filegroup and the time you restored it. This guarantees that the restored files are in sync
with the rest of the database.

For example, suppose you backed up Filegroup FG1 at 10 A.M. Monday. The database is still in use, and
changes happen to data on FG1 and transactions are processed that change data in both FG1 and other
filegroups. You back up the log at 4 P.M. More transactions are processed that change data in both FG1 and
other filegroups. At 6 P.M., a media failure occurs and you lose one or more of the files making up FG1.

To restore, you must first back up the tail of the log containing all changes that occurred between 4 P.M. and
6 P.M. You can then restore FG1 using the RESTORE DATABASE command, specifying just filegroup
FG1. Your database will not be in a consistent state because the restored FG1 will have changes only through
10 A.M. but the rest of the database will have changes through 6 P.M. However, SQL Server knows when
the last change was made to the database because each page in a database stores the LSN of the last log
record that changed that page. When restoring a filegroup, SQL Server makes a note of the maximum LSN in
the database. You must restore log backups until the log reaches at least the maximum LSN in the database,
and you will not reach that point until you apply the 6 P.M. log backup.

Partial restore
SQL Server 2000 lets you do a partial restore of a database in emergency situations. Although the description and
the syntax seem similar to file and filegroup backups, there is a big difference. With file and filegroup backups, you
start with a complete database and replace one or more files or filegroups with previously backed up versions. With a
partial database restore, you don't start with a full database. You restore individual filegroups, which must include the
primary filegroup containing all the system tables, to a new location. Any filegroups you don't restore no longer exist
and are treated as OFFLINE when you attempt to reference data stored on them. You can then restore log backups
or differential backups to bring the data in those filegroups to a later point in time. This allows you the option of
recovering the data from a subset of tables after an accidental deletion or modification of table data. You can use the
partially restored database to extract the data from the lost tables and copy it back into your original database.

Restoring with standby

In normal recovery operations, you have the choice of either running recovery to roll back incomplete transactions or
not running recovery. If you run recovery, no further log backups can be restored and the database is fully usable. If
you don't run recovery, the database is inconsistent and SQL Server won't let you use it at all. You have to choose
one or the other because of the way log backups are made.
This document is created with the unregistered version of CHM2PDF Pilot

For example, in SQL Server 2000, log backups do not overlap—each log backup starts where the previous one
ended. Consider a transaction that makes hundreds of updates to a single table. If you back up the log in the middle
of the updating and again after the updating is finished, the first log backup will have the beginning of the transaction
and some of the updates and the second log backup will have the remainder of the updates and the commit. Suppose
you then need to restore these log backups after restoring the full database. If, after restoring the first log backup, you
run recovery, the first part of the transaction is rolled back. If you then try to restore the second log backup, it will
start in the middle of a transaction and SQL Server won't know what the beginning of the transaction did. You
certainly can't recover transactions from this point because their operations might depend on this update that you've
lost part of. So, SQL Server will not allow any more restoring to be done. The alternative is to not run recovery to roll
back the first part of the transaction, but instead to leave the transaction incomplete. SQL Server will know that the
database is inconsistent and will not allow any users into the database until you finally run recovery on it.

What if you want to combine the two approaches? It would be nice to be able to restore one log backup and look at
the data before restoring more log backups, particularly if you're trying to do a point-in-time recovery, but you won't
know what the right point is. SQL Server provides an option called STANDBY that allows you to recover the
database and still restore more log backups. If you restore a log backup and specify WITH STANDBY = '<some
filename>', SQL Server will roll back incomplete transactions but keep track of the rolled-back work in the specified
file, which is known as a standby file. The next restore operation will first read the contents of the standby file and
redo the operations that were rolled back, and then it will restore the next log. If that restore also specifies WITH
STANDBY, incomplete transactions will again be rolled back but a record of those rolled back transactions will be
saved. Keep in mind that you can't modify any data if you've restored WITH STANDBY (SQL Server will generate
an error message if you try), but you can read the data and continue to restore more logs. The final log must be
restored WITH RECOVERY (and no standby file will be kept) to make the database fully usable.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
A database is a collection of objects such as tables, views, and stored procedures. Although a typical SQL Server
installation has many databases, it always includes the following three: master, model, and tempdb. (An installation
usually also includes pubs, Northwind, and msdb.) Every database has its own transaction log; integrity constraints
among objects keep a database logically consistent.

Databases are stored in operating system files in a one-to-many relationship. Each database has at least one file for
the data and one file for the transaction log. You can easily increase and decrease the size of databases and their files
either manually or automatically.

Now that you understand database basics, it's time to move on to tables, the fundamental data structures you'll work
with.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 6

Tables
In this chapter, I'll look at some in-depth implementation examples. But let's start with a basic introduction to tables.
Simply put, a table is a collection of data about a specific entity (person, place, or thing) that has a discrete number
of named attributes (for example, quantity or type). Tables are at the heart of Microsoft SQL Server and the
relational model in general. Tables are easy to understand—they're just like the everyday lists that you make for
yourself. In SQL Server, a table is often referred to as a base table to emphasize where data is stored. Calling it a
base table also distinguishes the table from a view, a virtual table that's an internal query referencing one or more base
tables.

Attributes of a table's data (such as color, size, quantity, order date, and supplier's name) take the form of named
columns in the table. Each instance of data in a table is represented as a single entry, or row (formally called a tuple).
In a true relational database, every row in a table is unique, and each row has a unique identifier called the primary
key. (SQL Server, in accordance with the ANSI SQL standard, doesn't require you to make a row unique or declare
a primary key. However, because both of these concepts are central to the relational model, you should always
implement them.)

Most tables will have some relationship to other tables. For example, in an order-entry system, the orders table likely
has a customer_number column in which it keeps track of the customer number for an order; customer_number
also appears in the customer table. Assuming that customer_number is a unique identifier, or primary key, of the
customer table, a foreign key relationship is established by which the orders and customer tables can subsequently
be joined.

So much for the 30-second database design primer. You can find plenty of books that discuss logical database and
table design, but this isn't one of them. I'll assume that you understand basic database theory and design and that you
generally know what your tables will look like. The rest of this chapter discusses the internals of tables in SQL Server
and implementation considerations.
This document is created with the unregistered version of CHM2PDF Pilot

34

Creating Tables
To create a table, SQL Server uses the ANSI SQL standard CREATE TABLE syntax. SQL Enterprise Manager
provides a front-end, fill-in-the-blanks table designer that can make your job easier. Ultimately, the SQL syntax is
always sent to SQL Server to create a table. You can create a table directly using a tool such as OSQL, ISQL, or
SQL Query Analyzer; from SQL Enterprise Manager; or using a third-party data-modeling tool (such as ERwin or
Microsoft Visual InterDev) that transmits the SQL syntax under the cover of a friendly interface.

In this chapter, I'll emphasize direct use of the data definition language (DDL) rather than discussing the interface
tools. You should keep all DDL commands in a script so that you can run them easily at a later time to re-create the
table. (Even if you use one of the friendly front-end tools, it's critical that you be able to re-create the table later.)
SQL Enterprise Manager and other front-end tools can create and save operating system files with the SQL DDL
commands necessary to create the object. This DDL is essentially source code, and you should treat it as such. Keep
a backup copy. You should also consider keeping these files under version control using a source control product
such as Microsoft Visual SourceSafe or PVCS from MERANT (formerly INTERSOLV).

At the basic level, creating a table requires little more than knowing what you want to name it, what columns it will
contain, and what range of values (domain) each column will be able to store. Here's the basic syntax for creating the
customer table, with three fixed-length character (char) columns. (Note that this table definition isn't necessarily the
most efficient way to store data because it always requires 46 bytes per entry plus a few bytes of overhead,
regardless of the actual length of the data.)
CREATE TABLE customer
(
name char(30),
phone char(12),
emp_id char(4)
)

This example shows each column on a separate line, for readability. As far as the SQL Server parser is concerned,
white spaces created by tabs, carriage returns, and the spacebar are identical. From the system's standpoint, the
following CREATE TABLE example is identical to the one above; but it's harder to read from a user's standpoint:
CREATE TABLE customer (name char(30), phone char(12), emp_id char(4))

We'll see many more detailed examples of tables later in the chapter.

Naming Tables and Columns

A table is always created within a database and is owned by one particular user. Normally, the owner is the user who
created the table, but anyone with the sysadmin, db_ddladmin, or db_owner role can create a table that is owned
by another user. A database can contain multiple tables with the same name, as long as the tables have different
owners. The full name of a table has three parts, in the following form:
database.owner.tablename

For example, say that a user (with the username Kalen) creates a sample customer table in the pubs sample
database. This user's table will have the pubs.kalen.customer three-part name. (If this user is also the database
This document is created with the unregistered version of CHM2PDF Pilot

owner, pubs.dbo.customer will be her table's name because dbo is the special username for the database owner in
every database.)

The first two parts of the three-part name specification have default values. The default for the name of the database
is whatever database context you're currently working in. The table owner actually has two possible defaults. If no
table owner name is specified when you reference a table, SQL Server assumes that either you or the owner of the
database owns the table. For example, if our hypothetical user owns the customer table and her database context is
pubs, she can refer to the table simply as customer.
NOTE
To access a table owned by anyone other than yourself or the database owner, you must include the owner name
along with the table name.

You should make column names descriptive, and because you'll use them repeatedly, you should avoid wordiness.
The name of the column (or any object in SQL Server, such as a table or a view) can be whatever you choose, as
long as it conforms to the SQL Server rule for identifiers: it must consist of a combination of 1 through 128 letters,
digits, or the symbols #, $, @, or _. (For more on identifier rules, see "Using Identifiers" in SQL Server Books
Online. The discussions there apply to all SQL Server object names, not just column names.)

In some cases, a table can be addressed using a four-part name, in which the first part is the name of the SQL
Server. However, you can refer to a table using a four-part name only if the SQL Server has been defined as a linked
server. You can read more about linked servers in SQL Server Books Online; I won't discuss them further here.

Reserved Keywords
Certain reserved keywords, such as table, create, select, and update, have special meaning to the SQL Server
parser, and collectively they make up the SQL language implementation. You should avoid using reserved keywords
for your object names. In addition to the SQL Server reserved keywords, the SQL-92 standard has its own list of
reserved keywords. In some cases, this list is more restrictive than SQL Server's list; in other cases, it's less
restrictive. SQL Server Books Online includes both lists.

Watch out for SQL-92's reserved keywords. Some of the words aren't reserved keywords in SQL Server yet, but
they might become reserved keywords in a future SQL Server version. If you use a SQL-92 reserved keyword, you
might end up having to alter your application before upgrading it if the word has become a SQL Server reserved
keyword.

Delimited Identifiers
You can't use keywords in your object names unless you use a delimited identifier. In fact, if you use a delimited
identifier, not only can you use keywords as identifiers but you can also use any other string as an object
name—whether or not it follows the rules for identifiers. This includes spaces and other nonalphanumeric characters
that are normally not allowed. Two types of delimited identifiers exist:
 Bracketed identifiers are delimited by square brackets ([object name]).
 Quoted identifiers are delimited by double quotation marks ("object name").

You can use bracketed identifiers in any environment, but to use quoted identifiers, you must enable a special option
using SET QUOTED_IDENTIFIER ON. If you turn on QUOTED_IDENTIFIER, double quotes are interpreted as
referencing an object. To delimit string or date constants, you must use single quotes.

Let's look at some examples. Because column is a reserved keyword, the first statement that follows is illegal in all
circumstances, and the second statement is illegal unless QUOTED_IDENTIFIER is on. The third statement is legal in
any circumstance.
CREATE TABLE customer(name char(30), column char(12), emp_id char(4))
This document is created with the unregistered version of CHM2PDF Pilot

CREATE TABLE customer(name char(30), "column" char(12), emp_id char(4))

CREATE TABLE customer(name char(30), [column] char(12), emp_id char(4))

The ODBC driver that comes with SQL Server sets the QUOTED_IDENTIFIER option to ON by default, but
some of the SQL Server-specific tools set it to OFF. You can determine whether this option is on or off for your
session by executing the following command:
DBCC USEROPTIONS

If you're using SQL Query Analyzer, you can check the setting by running the preceding command or by choosing
Options from the Tools menu and examining the Connection Properties tab.

Theoretically, you can always use delimited identifiers with all object and column names, so you never have to worry
about reserved keywords. However, I don't recommend this. Many third-party tools for SQL Server don't handle
quoted identifiers well. Using quoted identifiers might make upgrading to future versions of SQL Server more difficult.
For example, if you're using the SQL Server Upgrade Wizard to upgrade a database directly to SQL Server 2000
from SQL Server 6.5, objects such as views and stored procedures are automatically dropped and re-created so that
they include the structures of the latest version. If your tables have names that were reserved keywords, they have to
be re-created using quoted identifiers.

When you use quoted identifiers, upgrading can't be fully automated and you must tell the SQL Server Upgrade
Wizard whether it should assume that QUOTED_IDENTIFIER is on. If you don't know, or if you have a mixture of
objects, you can tell the wizard to use a mixed mode. It will then try to interpret all objects with double quotes as if
the option is on; if that's unsuccessful, it will assume that the option is off for that object. This interpretation might not
always result in the stored procedure or view having the same meaning as was intended.

Rather than using delimited identifiers to protect against reserved keyword problems, you should simply adopt some
simple naming conventions. For example, you can precede column names with the first few letters of the table name
and an underscore. This naming style makes the column or object name more readable and also greatly reduces your
chances of encountering a keyword or reserved word conflict.

Naming Conventions
Many organizations and multiuser development projects adopt standard naming conventions. This is generally good
practice. For example, assigning a standard moniker of cust_id to represent a customer number in every table clearly
shows that all the tables share common data. If an organization instead uses several monikers in the tables to represent
a customer number, such as cust_id, cust_num, customer_number, and customer_#, it won't be as obvious that
these monikers represent common data.

One naming convention is the Hungarian-style notation for column names. Hungarian-style notation is a widely used
practice in C programming, whereby variable names include information about their datatypes. This notation uses
names such as sint_nn_custnum to indicate that the custnum column is a small integer (smallint of 2 bytes) and is
NOT NULL (doesn't allow nulls). Although this practice makes good sense in C programming, it defeats the datatype
independence that SQL Server provides; therefore, I recommend against using it.

Suppose, for example, that after the table is built and applications have been written, you discover that the custnum
column requires a 4-byte integer (int) instead of a 2-byte small integer. You can use the ALTER TABLE command
to modify the datatype of the existing table to int instead of smallint. SQL Server stored procedures handle the
different datatypes automatically. Applications using ODBC, OLE DB, or DB-Library that bind the retrieved column
to a character or integer datatype will be unaffected. The applications would need to change if they bound the
column to a small integer variable because the variable's type would need to be larger. Therefore, you should try not
This document is created with the unregistered version of CHM2PDF Pilot

to be overly conservative with variable datatypes, especially in your client applications. You should be most
concerned with the type on the server side; the type in the application can be larger and will automatically
accommodate smaller values. By overloading the column name with datatype information, which is readily available
from the system catalogs, the insulation from the underlying datatype is compromised. If you changed the datatype
from a smallint to an int, the Hungarian-style name would no longer accurately reflect the column definition. Changing
the column name would then result in the need to change application code, stored procedures, or both.

Datatypes
SQL Server provides many datatypes, most of which I'll discuss in this section. Choosing the appropriate datatype is
simply a matter of mapping the domain of values you need to store to the corresponding datatype. In choosing
datatypes, you want to avoid wasting storage space while allowing enough space for a sufficient range of possible
values over the life of your application.

Datatype Synonyms
SQL Server syntactically accepts as datatypes both the words listed as synonyms and the base datatypes shown in
Table 6-1, but it uses only the type listed as the datatype. For example, you can define a column as character(1),
character, or char(1), and SQL Server will accept any of them as valid syntax. Internally, however, the expression is
considered char(1), and subsequent querying of the SQL Server system catalogs for the datatype will show it as
char(1) regardless of the syntax that you used when you created it.

Table 6-1. SQL Server datatype synonyms.

Synonym Mapped to system datatype
binary varying varbinary
char varying varchar
character char(1)
character(n) char(n)
character varying(n) varchar(n)
dec decimal
double precision float
float[(n)] for n = 1 to 7 real
float[(n)] for n = 8 to 15 float
integer int
national character(n) nchar(n)
national char(n) nchar(n)
national character varying(n) nvarchar(n)
national char varying(n) nvarchar(n)
national text ntext
numeric decimal

Choosing a Datatype
The decision about what datatype to use for each column depends primarily on the nature of the data the column will
hold. There are five basic datatypes in SQL Server 2000: numeric, character, datetime, LOB (large object), and
miscellaneous. SQL Server 2000 also supports a variant datatype called sql_variant. Values stored in a sql_variant
column can be of any datatype. I'll discuss LOB and sql_variant columns later in this chapter when I discuss the
internal storage of data on a page. I'll also provide more details about several of the datatypes later in the book when
I illustrate programming techniques using the various datatypes. In this section, I'll examine some of the issues related
This document is created with the unregistered version of CHM2PDF Pilot

to storing data of different datatypes.

Numeric Datatypes
You should use numeric datatypes for data on which you want to perform numeric comparisons or arithmetic
operations. Your main decisions are the maximum range of possible values you want to be able to store and the
accuracy you need. The tradeoff is that datatypes that can store a greater range of values take up more space.

Numeric datatypes can also be classified as either exact or approximate. Exact numeric values are guaranteed to
store exact representations of your numbers. Approximate numeric values have a far greater range of values, but the
values are not guaranteed to be stored precisely. Exact numeric values can store data in the range -10^38 + 1to
10^38 -1. Unless you need numbers with greater magnitude, I recommend that you don't use the approximate
numeric datatypes.

The exact numeric datatypes can be divided into two groups: integers and decimals. Integer types range in size from
1 to 8 bytes, with a corresponding increase in the range of possible values. The money and smallmoney datatypes
are frequently included among the integer types because internally they are stored in the same way. For the money
and smallmoney types, it is just understood that the rightmost four digits are after the decimal point. For the other
integer types, no digits come after the decimal point. Table 6-2 lists the integer datatypes along with their storage size
and range of values.

Table 6-2. SQL Server integer datatypes.

Datatype Range Storage (bytes)
bigint -263 to 263-1 8
int -231 to 231-1 4
smallint -215 to 215-1 2
tinyint 0 to 255 1
- 922,337,203,685,477.5808 to
922,337,203,685,477.5807 with
money 8
accuracy of one ten-thousandth of a
monetary unit
-214,748.3648 to 214,748.3647,
smallmoney with accuracy of one ten-thousandth 4
of a monetary unit

The decimal and numeric datatypes allow quite a high degree of accuracy as well as a large range of values For these
two synonymous datatypes, you can specify a precision (the total number of digits stored) and a scale (the maximum
number of digits to the right of the decimal point). The maximum number of digits that can be stored to the left of the
decimal point is (precision - scale). Two different decimal values can have the same precision and very different
ranges. For example, a column defined as decimal (8,4) can store values from -9999.9999 to 9999.9999 and a
column defined as decimal (8,0) can store values from -99,999,999 to 99,999,999.

Table 6-3 shows the storage space required for decimal and numeric data based on the defined precision.

Table 6-3. SQL Server decimal and numeric datatype storage requirements.
Precision Storage (bytes)
1 to 9 5
10 to 19 9
20 to 28 13
29 to 38 17
This document is created with the unregistered version of CHM2PDF Pilot

Datetime Datatypes
SQL Server supports two datatypes for storing date and time information: datetime and smalldatetime. Again, the
difference between these types is the range of possible dates and the number of bytes needed for storage. Both types
have both a date and a time component. The range and storage requirements are shown in Table 64. If no date is
supplied, the default of January 1, 1900 is assumed; if no time is supplied, the default of 00:00:00.000 (midnight) is
assumed. In Chapter 7, I'll address many issues dealing with how SQL Server interprets datetime values entered by
an application and how you can control the formatting of the displayed datetime value.

Internally, datetime and smalldatetime values are stored completely differently from how you enter them or how they
are displayed. They are stored as two separate components, a date component and a time component. The date is
stored as the number of days before or after the base date of January 1, 1900. For datetime values, the time is stored
as the number of clock ticks after midnight, with each tick representing 3.33 milliseconds, or 1/300 of a second. For
smalldatetime values, the time is stored as the number of minutes after midnight. You can actually see these two parts
if you convert a datetime value to a binary string of 8 hexadecimal bytes. The first four hexadecimal bytes are the
number of days before or after the base date, and the second four bytes are the number of clock ticks after midnight.
You can then convert these four-byte hexadecimal strings to integers. I'll provide more details about hexadecimal
strings later in the chapter.

The following example shows how to see the component parts of the current date and time, which is available using
the parameterless system function current_timestamp. The example stores the current date and time in a local variable
so we can know we're using the same value for both computations. I'll talk about local variables in detail in Chapter
10.
DECLARE @today datetime
SELECT @today = current_timestamp
SELECT @today
SELECT CONVERT (varbinary(8), @today)
SELECT CONVERT (int, SUBSTRING (CONVERT (varbinary(8), @today), 1, 4))
SELECT CONVERT (int, SUBSTRING (CONVERT (varbinary(8), @today), 5, 4))

Table 6-4. SQL Server datetime datatypes.

Datatype Range Storage (bytes)
January 1, 1753, through December
datetime 31, 9999, with an accuracy of 8
three-hundredths of a second
January 1, 1900, through June 6,
smalldatetime 4
2079,with an accuracy of one minute

Character Datatypes
Character datatypes come in four varieties. They can be fixed-length or variable-length single-byte character strings (
char and varchar) or fixed-length or variable-length Unicode character strings (nchar and nvarchar). Unicode
character strings need two bytes for each stored character; use them when you need to represent characters that can't
be stored in the single-byte characters that are sufficient for storing most of the characters in the English and European
alphabets. Single-byte character strings can store up to 8000 characters, and Unicode character strings can store up
to 4000 characters. You should know the type of data you'll be dealing with in order to decide between single-byte
and double-byte character strings.

Deciding to use a variable-length or a fixed-length datatype is a more difficult decision, and it isn't always
straightforward or obvious. As a general rule, variable-length datatypes are most appropriate when you expect
significant variation in the size of the data for a column and when the data in the column won't be frequently changed.
This document is created with the unregistered version of CHM2PDF Pilot

Using variable-length datatypes can yield important storage savings. They can sometimes result in a minor
performance loss, and at other times they can result in improved performance. A row with variable-length columns
requires special offset entries in order to be internally maintained. These entries keep track of the actual length of the
column. Calculating and maintaining the offsets requires slightly more overhead than does a pure fixed-length row,
which needs no such offsets. This task requires a few addition and subtraction operations to maintain the offset value.
However, the extra overhead of maintaining these offsets is generally inconsequential, and this alone would not make a
significant difference on most systems, if any.

Another potential performance issue with variable length fields is the cost of increasing the size of a row on an almost
full page. If a row with variable-length columns uses only part of its maximum length and is later updated to a longer
length, the enlarged row might no longer fit on the same page. If the table has a clustered index, the row must stay in
the same position relative to the other rows, so the solution is to split the page and move some of the rows from the
page with the enlarged row onto a newly linked page. This can be an expensive operation. If the table has no
clustered index, the row can move to a new location and leave a forwarding pointer in the original location. I'll talk
about the details of page splitting and moving rows when I discuss data modification operations in Chapter 9.

On the other hand, using variable-length columns can sometimes improve performance because they can allow more
rows to fit on a page. But the efficiency results from more than simply requiring less disk space. A data page for SQL
Server is 8 KB (8192 bytes), of which 8096 bytes are available to store data. (The rest is for internal use to keep
track of structural information about the page and the object to which it belongs.) One I/O operation brings back the
entire page. If you can fit 80 rows on a page, a single I/O operation brings back 80 rows. But if you can fit 160 rows
on a page, one I/O operation is essentially twice as efficient. In operations that scan for data and return lots of
adjacent rows, this can amount to a significant performance improvement. The more rows you can fit per page, the
better your I/O and cache-hit efficiency will be.

For example, consider a simple customer table. Suppose you could define it in two ways, fixed-length and
variable-length, as shown in Figures 6-1 and 6-2.

Figure 6-1. A customer table with all fixed-length columns.

This document is created with the unregistered version of CHM2PDF Pilot

Figure 6-2. A customer table with variable-length columns.

Columns that contain addresses, names, or URLs all have data that varies significantly in length. Let's look at the
differences between choosing fixed-length columns vs. choosing variable-length columns. In Figure 6-1, which uses all
fixed-length columns, every row uses 304 bytes for data regardless of the number of characters actually inserted in the
row. SQL Server also needs an additional 10 bytes of overhead for every row in this table, so each row needs a total
of 314 bytes for storage. But let's say that even though the table must accommodate addresses and names up to the
specified size, the average entry is only half the maximum size.

In Figure 6-2, assume that for all the variable-length (varchar) columns, the average entry is actually only about half
the maximum. Instead of a row length of 304 bytes, the average length is 184 bytes. This length is computed as
follows: the smallint and char(2) columns total 4 bytes. The varchar columns' maximum total length is 300, half of
which is 150 bytes. And a 2-byte overhead exists for each of nine varchar columns, for 18 bytes. Add 2 more bytes
for any row that has one or more variable-length columns. In addition, such rows require another 10 bytes of
overhead, regardless of the presence of variable-length fields. (This is the same 10 bytes of overhead needed in the
case of all fixed-length columns; in other words, all rows have these same 10 bytes of constant overhead.) So the total
is 4 + 150 + 18 + 2 + 10, or 184. (I'll discuss the actual meaning of each of these bytes of overhead later in this
chapter.)

In the fixed-length example in Figure 6-1, you always fit 25 rows on a data page (8096/314, discarding the
remainder). In the variable-length example in Figure 6-2, you can fit an average of 44 rows per page (8096/184). The
table using variable-length columns will consume about half as many pages in storage, a single I/O operation will
retrieve almost twice as many rows, and a page cached in memory is twice as likely to contain the row you're looking
for.

When choosing lengths for columns, don't be wasteful—but don't be cheap, either. Allow for future needs, and
realize that if the additional length doesn't change how many rows will fit on a page, the additional size is free anyway.
Consider again the examples in Figures 6-1 and 6-2. The cust_id was declared as a smallint, meaning that its
maximum positive value is 32,767 (unfortunately, SQL Server doesn't provide any unsigned int or unsigned smallint
datatypes), and it consumes 2 bytes of storage. Although 32,767 customers might seem like a lot to a new company,
the company might be surprised by its own success and, in a couple of years, find out that 32,767 is too limited.

The database designers might regret that they tried to save 2 bytes and didn't simply make the datatype an int, using
4 bytes but with a maximum positive value of 2,147,483,647. They'll be especially disappointed if they realize they
didn't really save any space. If you compute the rows-per-page calculations just discussed, increasing the row size by
2 bytes, you'll see that the same number of rows still fit on a page. The additional 2 bytes are free—they were simply
wasted space before. They never cause fewer rows per page in the fixed-length example, and they'll rarely cause
fewer rows per page even in the variable-length case.
This document is created with the unregistered version of CHM2PDF Pilot

So which strategy wins? Potentially better update performance? Or more rows per page? Like most questions of this
nature, no one answer is right. It depends on your application. If you understand the tradeoffs, you'll be able to make
the best choice. Now that you know the issues, this general rule merits repeating: variable-length datatypes are most
appropriate when you expect significant variation in the size of the data for that column and when the column won't be
updated frequently.

Miscellaneous Datatypes
I'll end this part of the discussion by showing you a few additional datatypes that you might have use for.

Binary These datatypes are binary and varbinary. They are used to store strings of bits, and the values are entered
and displayed using their (hex) hexadecimal representation, which is indicated by a prefix of 0x. So a hex value of
0x270F corresponds to a decimal value of 9999 and a bit string of 10011100001111. In hex, each two displayed
characters represent a byte, so the value of 0x270F represents 2 bytes. You need to decide whether you want your
data to be fixed or variable length, and you can use some of the same considerations I discussed for deciding
between char and varchar to make your decision. The maximum length of binary or varbinary data is 8000 bytes.

Bit The bit datatype can store a 0 or a 1 and can consume only a single bit of storage space. However, if there is
only one bit column in a table, it will take up a whole byte. Up to 8 bit columns are stored in a single byte.

Large Object These datatypes are text, ntext, and image. The text datatype can store up to 2^31 - 1
non-Unicode characters, ntext can store up to 2^30 -1 (half as many) Unicode characters, and image can store up
to 2^31 - 1 bytes of binary data. We'll look at the storage mechanisms used for these large object (LOB) datatypes
later in the chapter.

Cursor The cursor datatype can hold a reference to a cursor. Although you can't declare a column in a table to be
of type cursor, I've included cursor in this list for completeness. I'll talk about cursors and cursor variables in detail in
Chapter 13.

Rowversion The rowversion datatype is a synonym in SQL Server 2000 for what was formerly called a
timestamp. I like the new name because people always assumed that the timestamp datatype had something to do
with dates or times, and it doesn't. A column of type rowversion holds an internal sequence number that SQL Server
automatically updates every time the row is modified. The value of any rowversion column is actually unique within an
entire database, and a table can have only one column of type rowversion. Any operation that modifies any
rowversion column in the database generates the next sequential value. The actual value stored in a rowversion
column is seldom important by itself. The column is used to detect whether a row has been modified since the last time
it was accessed by finding out whether the rowversion value has changed.

Sql_variant The sql_variant datatype allows a column to hold values of any other datatype except text, ntext,
image, or rowversion (timestamp). I'll describe the internal storage of sql_variant data later in this chapter, and I'll
show you some examples of using sql_variant data in Chapter 7.

Table The table datatype can be used to store the result of a function and can be used as the datatype of local
variables. Columns in tables cannot be of type table. I'll talk about the table datatype in more detail in Chapter 7
when I discuss user-defined functions.

Uniqueidentifier The uniqueidentifier datatype is sometimes referred to as a globally unique identifier (GUID) or
universal unique identifier (UUID). A GUID or UUID is a 128-bit (16-byte) value generated in a way that, for all
practical purposes, guarantees uniqueness worldwide, even among unconnected computers. It is becoming an
important way to identify data, objects, software applications, and applets in distributed systems. Since I won't be
talking about uniqueidentifier data anywhere else in the book, I'll give you a bit more detail about it here.

The Transact-SQL language supports the system function NEWID, which can be used to generate a
uniqueidentifier value. A column or variable of datatype uniqueidentifier can be initialized to a value in one of the
This document is created with the unregistered version of CHM2PDF Pilot

following two ways:

 Using the system-supplied function NEWID.
 Using a string constant in the following form (32 hexadecimal digits separated by hyphens):
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. (Each x is a hexadecimal digit in the range 0 through 9 or a
through f.)

This datatype can be quite cumbersome to work with, and the only operations that are allowed against a
uniqueidentifier value are comparisons (=, <>, <, >, <=, >=) and checking for NULL. However, using this datatype
internally can offer several advantages.

One advantage is that the values are guaranteed to be globally unique for any machine on a network because the last
six bytes of a uniqueidentifier value make up the node number for the machine. On a machine without a NIC (such
as a home computer that connects to the Internet via modem), the node is a pseudorandom 48-bit value that isn't
guaranteed to be unique now but is highly likely to be unique in the near future.

Another advantage is that the list of uniqueidentifier values can't be exhausted. This is not the case with other
datatypes frequently used as unique identifiers. In fact, SQL Server uses this datatype internally by for row-level
merge replication. A uniqueidentifier column can have a special property called the ROWGUIDCOL property; at
most, one uniqueidentifier column can have this property per table. The ROWGUIDCOL property can be specified
as part of the column definition in CREATE TABLE and ALTER TABLE ADD column or can be added or dropped
for an existing column using ALTER TABLE ALTER COLUMN.

You can reference a uniqueidentifier column with the ROWGUIDCOL property using the keyword
ROWGUIDCOL in a query. This is similar to referencing an identity column using the IDENTITYCOL keyword.
The ROWGUIDCOL property does not imply any automatic value generation, and if automatic value generation is
needed, the NEWID function should be defined as the default value of the column. You can have multiple
uniqueidentifier columns per table, but only one of them can have the ROWGUIDCOL property. You can use the
uniqueidentifier datatype for whatever reason you come up with, but if you're using one to identify the current row,
an application must have a generic way to ask for it without needing to know the column name. That's what the
ROWGUIDCOL property does.

Much Ado About NULL

The issue of whether to allow NULL has become an almost religious one for many in the industry, and no doubt the
discussion here will outrage a few people. However, my intention isn't to engage in a philosophical debate.
Pragmatically, dealing with NULL brings added complexity to the storage engine because SQL Server keeps a
special bitmap in every row to indicate which nullable columns actually are NULL. If NULLs are allowed, SQL
Server must decode this bitmap for every row accessed. Allowing NULL also adds complexity in application code,
which can often lead to bugs. You must always add special logic to account for the case of NULL.

As the database designer, you might understand the nuances of NULL and three-valued logic in aggregate functions,
when doing joins, and when searching by values. However, you must consider whether your development staff also
understands. I recommend, if possible, that you use all NOT NULL columns and define default values (discussed
later in this chapter) for missing or unknown entries (and possibly make such character columns varchar if the default
value is significantly different in size from the typical entered value).

In any case, it's good practice to explicitly declare NOT NULL or NULL when creating a table. If no such
declaration exists, SQL Server assumes NOT NULL. (In other words, no NULLs are allowed.) However, you can
set the default to allow NULLs by using a session setting or a database option. The ANSI SQL standard says that if
neither is specified, NULL should be assumed, but as I mentioned, this isn't SQL Server's default. If you script your
DDL and then run it against another server that has a different default setting, you'll get different results if you don't
explicitly declare NULL or NOT NULL in the column definition.

Several database options and session settings can control SQL Server's behavior regarding NULL values. You can
This document is created with the unregistered version of CHM2PDF Pilot

set database options using the ALTER DATABASE command, as I showed you in Chapter 5. And you can enable
session settings for one connection at a time using the SET command.
NOTE
The database option ANSI null default corresponds to the two session settings ANSI_NULL_DFLT_ON or
ANSI_NULL_DFLT_OFF. When the ANSI null default database option is false (the default setting for SQL
Server), new columns created with the ALTER TABLE and CREATE TABLE statements are, by default, NOT
NULL if the nullability status of the column isn't explicitly specified. SET ANSI_NULL_DFLT_OFF and SET
ANSI_NULL_DFLT_ON are mutually exclusive options that indicate whether the database option should be
overridden. When on, each option forces the opposite option off. Neither option, when set off, turns the opposite
option on—it only discontinues the current on setting.

You use the function GETANSINULL to determine the default nullability for your current session. This function
returns 1 when new columns allow null values and the column or datatype nullability wasn't explicitly defined when the
table was created or altered. I strongly recommend declaring NULL or NOT NULL explicitly when you create a
column. This removes all ambiguity and ensures that you're in control of how the table will be built, regardless of the
default nullability setting.
WARNING
Although internally SQL Server's default behavior is to not allow NULLs unless they are specifically declared in the
CREATE TABLE statement, you might never see this behavior in action. Because SQL Query Analyzer—the basic
tool for submitting SQL code to SQL Server—is ODBC-based, it automatically turns on the
ANSI_NULL_DFLT_ON option. This setting means that all your new columns will allow NULLs by default. I can't
overemphasize that your best bet for avoiding confusion is to always state explicitly in your table definition whether
NULLs should be allowed.

The database option concat null yields null corresponds to the session setting SET
CONCAT_NULL_YIELDS_NULL. When CONCAT_NULL_YIELDS_NULL is on, concatenating a NULL
value with a string yields a NULL result. For example, SELECT 'abc' + NULL yields NULL. When SET
CONCAT_NULL_YIELDS_NULL is off, concatenating a NULL value with a string yields the string itself. In other
words, the NULL value is treated as an empty string. For example, SELECT 'abc' + NULL yields abc. If the session
level setting isn't specified, the value of the database option concat null yields null applies.

The database option ANSI nulls corresponds to the session setting SET ANSI_NULLS. When true, all comparisons
to a null value evaluate to FALSE ). When this option is set to false, comparisons of non-Unicode values to a null
value evaluate to TRUE if both values are NULL. In addition, when this option is set to true, your code must use the
condition IS NULL to determine whether a column has a NULL value. When this option is set to false, SQL Server
allows "= NULL" as a synonym for "IS NULL" and "<> NULL" as a synonym for "IS NOT NULL."

You can see this behavior yourself by looking at the titles table in the pubs database. The titles table has two rows
with a NULL price. The first batch of statements that follows, when executed from SQL Query Analyzer, should
return two rows, and the second batch should return no rows.
-- First batch will return 2 rows
USE pubs
SET ANSI_NULLS OFF
GO
SELECT * FROM titles WHERE price = NULL
GO

-- Second batch will return no rows

USE pubs
SET ANSI_NULLS ON
GO
SELECT * FROM titles WHERE price = NULL
GO
This document is created with the unregistered version of CHM2PDF Pilot

A fourth session setting is ANSI_DEFAULTS. Setting this to ON is a shortcut for enabling both ANSI_NULLS and
ANSI_NULL_DFLT_ON as well as other session settings not related to NULL handling. SQL Server's ODBC
driver and the SQL Server OLE DB provider automatically set ANSI_DEFAULTS to ON. You can change the
ANSI_NULLS setting when you define your data source name (DSN). Two of the client tools supplied with SQL
Server (SQL Query Analyzer and the text-based OSQL) use the SQL Server ODBC driver but then internally turn
off some of these options. To see which options are enabled for the tool you're using, you can run the following
command:
DBCC USEROPTIONS

Here's a sample of the output it might return:

Set Option Value
-------------------------- -------------------------
textsize 64512
language us_english
dateformat mdy
datefirst 7
ansi_null_dflt_on SET
ansi_warnings SET
ansi_padding SET
ansi_nulls SET
concat_null_yields_null SET

WARNING
The relationship between the database options and the corresponding session level SET options is not clearly
documented for the concat null yields null and ANSI nulls options. The SET option always overrides the database
option, whether the SET option has been turned ON or OFF. The only time the database option applies is when the
SET option has not been touched. So you can think of the ANSI_NULLS and CONCAT_NULL_YIELDS_NULL
options as having three possible values: ON, OFF and "never been set." If the SET option is either ON or OFF, the
database option is ignored. Since the ODBC driver (used by both SQL Query Analyzer and the OSQL command
tool) as well as the OLE DB provider for SQL Server set both of these options to ON, the database option is never
taken into account.

In addition, the command DBCC USEROPTIONS shows you which options have been set to ON, but it doesn't
show you which options have been explicitly set to OFF. I recommend that you try to write your queries so that it
doesn't matter which option is in effect. When that is impossible, you should rely on the session-level option only and
not the database-level option.

The database compatibility level controls two additional aspects of how SQL Server handles NULL values, as
determined by the system procedure sp_dbcmptlevel. If the compatibility level is set to 70 or 80, the nullability of bit
columns without explicit nullability is determined by either the session setting of SET ANSI_NULL_DFLT_ON or
SET ANSI_NULL_DFLT_OFF or the database setting of ANSI null default. In 60 or 65 compatibility mode, bit
columns created without an explicit NULL or NOT NULL option in CREATE TABLE or ALTER TABLE are
created as NOT NULL.

The database compatibility level also controls whether SQL Server interprets an empty string (two single quotes with
nothing between them) as either a single space or a true empty string. In compatibility level 60 or 65, SQL Server
interprets empty strings as single spaces. If the compatibility level is 70 or 80, SQL Server interprets empty strings as
truly empty—that is, a character string with no characters in it. Sometimes this empty string is referred to as a NULL,
but SQL Server doesn't treat it like a NULL. SQL Server marks NULLs internally as NULLs, but an empty string is
actually stored as a variable-length character field of 0 length.

In 60 or 65 compatibility mode, the empty string is interpreted as a single space in INSERT or assignment statements
on varchar data. When you are concatenating varchar, char, or text data, the empty string is interpreted as a single
space. This means that you can never have a truly empty string. The only alternative in 60 or 65 compatibility mode is
This document is created with the unregistered version of CHM2PDF Pilot

to define the field as allowing NULLs and to use a NULL in place of an empty string.

As you can see, you can configure and control the treatment and behavior of NULL values in several ways, and you
might think it would be impossible to keep track of all the variations. If you try to control every aspect of NULL
handling separately within each individual session, you can cause immeasurable confusion and even grief. However,
most of the issues become moot if you follow a few basic recommendations:
 Never allow NULL values in your tables.
 Include a specific NOT NULL qualification in your table definitions.
 Make sure all your databases are running in 80 compatibility mode.

If you must use NULLs in some cases, you can minimize problems by always following the same rules, and the easiest
rules to follow are the ones that ANSI already specifies.
This document is created with the unregistered version of CHM2PDF Pilot

34

User-Defined Datatypes
A user-defined datatype (UDDT) provides a convenient way for you to guarantee consistent use of underlying native
datatypes for columns known to have the same domain of possible values. For example, perhaps your database will
store various phone numbers in many tables. Although no single, definitive way exists to store phone numbers, in this
database consistency is important. You can create a phone_number UDDT and use it consistently for any column in
any table that keeps track of phone numbers to ensure that they all use the same datatype. Here's how to create this
UDDT:
EXEC sp_addtype phone_number, 'varchar(20)', 'not null'

And here's how to use the new UDDT when you create a table:
CREATE TABLE customer
(
cust_id smallint NOT NULL,
cust_name varchar(50) NOT NULL,
cust_addr1 varchar(50) NOT NULL,
cust_addr2 varchar(50) NOT NULL,
cust_city varchar(50) NOT NULL,
cust_state char(2) NOT NULL,
cust_zip varchar(10) NOT NULL,
cust_phone phone_number,
cust_fax varchar(20) NOT NULL,
cust_email varchar(30) NOT NULL,
cust_web_url varchar(20) NOT NULL
)

When the table is created, internally the cust_phone datatype is known to be varchar(20). Notice that both
cust_phone and cust_fax are varchar(20), although cust_phone has that declaration through its definition as a
UDDT.

Here's how the customer table appears in the entries in the syscolumns table for this table:
SELECT colid, name, xtype, length, xusertype, offset
FROM syscolumns WHERE id=object_id('customer')

colid name xtype length xusertype offset

------ ------------ ----- ------ --------- ------
1 cust_id 52 2 52 2
2 cust_name 167 50 167 -1
3 cust_addr1 167 50 167 -2
4 cust_addr2 167 50 167 -3
5 cust_city 167 50 167 -4
6 cust_state 175 2 175 4
7 cust_zip 167 10 167 -5
8 cust_phone 167 20 261 -6
9 cust_fax 167 20 167 -7
10 cust_email 167 30 167 -8
11 cust_web_url 167 20 167 -9

You can see that both the cust_phone and cust_fax columns have the same xtype (datatype), although the
This document is created with the unregistered version of CHM2PDF Pilot

cust_phone column shows that the datatype is a UDDT (xusertype = 261). The type is resolved when the table is
created, and the UDDT can't be dropped or changed as long as one or more tables are currently using it. Once
declared, a UDDT is static and immutable, so no inherent performance penalty occurs in using a UDDT instead of the
native datatype.

The use of UDDTs can make your database more consistent and clear. SQL Server implicitly converts between
compatible columns of different types (either native types or UDDTs of different types).

Currently, UDDTs don't support the notion of subtyping or inheritance, nor do they allow a DEFAULT value or
CHECK constraint to be declared as part of the UDDT itself. These powerful object-oriented concepts will likely
make their way into future versions of SQL Server. These limitations notwithstanding, UDDT functionality is a
dynamic and often underused feature of SQL Server.
This document is created with the unregistered version of CHM2PDF Pilot

34

Identity Property
It is common to provide simple counter-type values for tables that don't have a natural or efficient primary key.
Columns such as cust_id are usually simple counter fields. The IDENTITY property makes generating unique
numeric values easy. IDENTITY isn't a datatype; it's a column property that you can declare on a whole-number
datatype such as tinyint, smallint, int, or numeric/decimal (having a scale of zero). Each table can have only one
column with the IDENTITY property. The table's creator can specify the starting number (seed) and the amount that
this value increments or decrements. If not otherwise specified, the seed value starts at 1 and increments by 1, as
shown in this example:
CREATE TABLE customer
(
cust_id smallint IDENTITY NOT NULL,
cust_name varchar(50) NOT NULL
)

To find out which seed and increment values were defined for a table, you can use the IDENT_SEED(tablename)
and IDENT_INCR(tablename) functions. The statement
SELECT IDENT_SEED('customer'), IDENT_INCR('customer')

produces
1 1

for the customer table because values weren't explicitly declared and the default values were used.

This next example explicitly starts the numbering at 100 (seed) and increments the value by 20:
CREATE TABLE customer
(
cust_id smallint IDENTITY(100, 20) NOT NULL,
cust_name varchar(50) NOT NULL
)

The value automatically produced with the IDENTITY property is normally unique, but that isn't guaranteed by the
IDENTITY property itself. Nor is it guaranteed to be consecutive. For efficiency, a value is considered used as soon
as it is presented to a client doing an INSERT operation. If that client doesn't ultimately commit the INSERT, the
value never appears, so a break occurs in the consecutive numbers. An unacceptable level of serialization would exist
if the next number couldn't be parceled out until the previous one was actually committed or rolled back. (And even
then, as soon as a row was deleted, the values would no longer be consecutive. Gaps are inevitable.)
NOTE
If you need exact sequential values without gaps, IDENTITY isn't the appropriate feature to use. Instead, you should
implement a next_number-type table in which you can make the operation of bumping the number contained within it
part of the larger transaction (and incur the serialization of queuing for this value).

To temporarily disable the automatic generation of values in an identity column, you use the SET
IDENTITY_INSERT tablename ON option. In addition to filling in gaps in the identity sequence, this option is useful
This document is created with the unregistered version of CHM2PDF Pilot

for tasks such as bulk-loading data in which the previous values already exist. For example, perhaps you're loading a
new database with customer data from your previous system. You might want to preserve the previous customer
numbers but have new ones automatically assigned using IDENTITY. The SET option was created exactly for cases
like this.

Because of the SET option's ability to allow you to determine your own values for an IDENTITY column, the
IDENTITY property alone doesn't enforce uniqueness of a value within the table. Although IDENTITY will generate
a unique number if IDENTITY_INSERT has never been enabled, the uniqueness is not guaranteed once you have
used the SET option. To enforce uniqueness (which you'll almost always want to do when using IDENTITY), you
should also declare a UNIQUE or PRIMARY KEY constraint on the column. If you insert your own values for an
identity column (using SET IDENTITY_INSERT), when automatic generation resumes, the next value is the next
incremented value (or decremented value) of the highest value that exists in the table, whether it was generated
previously or explicitly inserted.
TIP
If you're using the bcp utility for bulk loading data, be aware of the -E (uppercase) parameter if your data already has
assigned values that you want to keep for a column that has the IDENTITY property. You can also use the
Transact-SQL BULK INSERT command with the KEEPIDENTITY option. For more information, see the SQL
Server documentation for bcp and BULK INSERT.

The keyword IDENTITYCOL automatically refers to the specific column in a table that has the IDENTITY property,
whatever its name. If that column is cust_id, you can refer to the column as IDENTITYCOL without knowing or
using the column name or you can refer to it explicitly as cust_id. For example, the following two statements work
identically and return the same data:
SELECT IDENTITYCOL FROM customer
SELECT cust_id FROM customer

The column name returned to the caller is cust_id, not IDENTITYCOL, in both of these cases.

When inserting rows, you must omit an identity column from the column list and VALUES section. (The only
exception is when the IDENTITY_INSERT option is on.) If you do supply a column list, you must omit the column
for which the value will be automatically supplied. Here are two valid INSERT statements for the customer table
shown earlier:
INSERT customer VALUES ('ACME Widgets')
INSERT customer (cust_name) VALUES ('AAA Gadgets')

Selecting these two rows produces this output:

cust_id cust_name
------- ---------
1 ACME Widgets
2 AAA Gadgets

(2 row(s) affected)

Sometimes in applications, it's desirable to immediately know the value produced by IDENTITY for subsequent use.
For example, a transaction might first add a new customer and then add an order for that customer. To add the order,
you probably need to use the cust_id. Rather than selecting the value from the customer table, you can simply select
the special system function @@IDENTITY, which contains the last identity value used by that connection. It doesn't
necessarily provide the last value inserted in the table, however, because another user might have subsequently
inserted data. If multiple INSERT statements are carried out in a batch on the same or different tables, the variable
has the value for the last statement only. In addition, if there is an INSERT trigger that fires after you insert the new
row and if that trigger inserts rows into a table with an identity column, @@IDENTITY will not have the value
This document is created with the unregistered version of CHM2PDF Pilot

inserted by the original INSERT statement. To you, it might look like you're inserting and then immediately checking
the value:
INSERT customer (cust_name) VALUES ('AAA Gadgets')
SELECT @@IDENTITY

However, if a trigger was fired for the INSERT, the value of @@IDENTITY might have changed.

There are two other functions that you might find useful when working with identity columns. SCOPE_IDENTITY
returns the last identity value inserted into a table in the same scope, which could be a stored procedure, trigger, or
batch. So if we replace @@IDENTITY with the SCOPE_IDENTITY function in the code snippet above, we can
see the identity value inserted into the customer table. If an INSERT trigger also inserted a row that contained an
identity column, it would be in a different scope:
INSERT customer (cust_name) VALUES ('AAA Gadgets')
SELECT SCOPE_IDENTITY()

In other cases, you might want to know the last identity value inserted in a specific table, from any application or user.
You can get this value using the IDENT_CURRENT function, which takes a table name as an argument:
SELECT IDENT_CURRENT('customer')

This doesn't always guarantee that you can predict the next identity value to be inserted, because another process
could insert a row between the time you check the value of IDENT_CURRENT and the time you execute your
INSERT statement.

You can't define the IDENTITY property as part of a UDDT, but you can declare the IDENTITY property on a
column that uses a UDDT. A column that has the IDENTITY property must always be declared NOT NULL (either
explicitly or implicitly); otherwise, error number 8147 will result from the CREATE TABLE statement and CREATE
won't succeed. Likewise, you can't declare the IDENTITY property and a DEFAULT on the same column. To
check that the current identity value is valid based on the current maximum values in the table, and to reset it if an
invalid value is found (which should never be the case), use the DBCC CHECKIDENT(tablename) statement.

Identity values are fully recoverable. If a system outage occurs while insert activity is taking place with tables that
have identity columns, the correct value will be recovered when SQL Server is restarted. SQL Server accomplishes
this during the checkpoint processing by flushing the current identity value for all tables. For activity beyond the last
checkpoint, subsequent values are reconstructed from the transaction log during the standard database recovery
process. Any inserts into a table that have the IDENTITY property are known to have changed the value, and the
current value is retrieved from the last INSERT statement (post-checkpoint) for each table in the transaction log. The
net result is that when the database is recovered, the correct current identity value is also recovered.

In rare cases, the identity value can get out of sync. If this happens, you can use the DBCC CHECKIDENT
command to reset the identity value to the appropriate number. In addition, the RESEED option to this command
allows you to set a new starting value for the identity sequence. See the online documentation for complete details.
This document is created with the unregistered version of CHM2PDF Pilot

34

Internal Storage
This section covers system catalogs and the internal data storage of tables. Although you can use SQL Server
effectively without understanding the internals, understanding the details of how SQL Server stores data will help you
develop efficient applications. (If you don't need this in-depth information, you can skip this discussion and proceed to
the section on constraints.)

When you create a table, one or more rows are inserted into a number of system catalogs to manage that table. At a
minimum, rows are added to the sysobjects, sysindexes, and syscolumns system catalogs (tables). When you define
the new table with one or more foreign key constraints, rows are added to the sysreferences system table.

For every table created, a single row that contains—among other things—the name, object ID, and owner of the
new table is added to the sysobjects table. The syscolumns table gains one row for each column in the new table,
and each row will contain information such as the column name, datatype, and length. Each column receives a column
ID, which directly corresponds to the order in which you specified the columns when you created the table. That is,
the first column listed in the CREATE TABLE statement will have a column ID of 1, the second column will have a
column ID of 2, and so on. Figure 6-3 shows the rows added to the sysobjects, and syscolumns system tables when
you create a table. (Not all columns are shown for each table.)

Figure 6-3. Catalog information stored after a table is created.

Notice in the syscolumns output in the figure that the xoffset column contains negative numbers in some rows. Any
column that contains variable-length data will have a negative xoffset value in syscolumns. Successive variable-length
columns in the CREATE TABLE statement are given decreasing xoffset values (-1, -2, -3, and so on). You can see
in Figure 6-3 that the employee last name (emp_lname) is the first variable-length column in the table.

The sysindexes table will have at least one row that will eventually contain pointers to the storage space that the new
table uses and information about the table's size. However, when the table is first created, no space is allocated to it,
so the columns in sysindexes that indicate page addresses and storage space reserved will all be 0. Note that this is a
change from all previous versions of SQL Server, in which at least two pages were allocated to the table as soon as it
was created—one page for the data, one root page for each index, one Index Allocation Map (IAM) page for each
index, and one IAM page for the table itself.

In SQL Server 2000, no space is allocated until the first row is inserted. If the table is defined without any primary
key or unique constraints, sysindexes will have only one row, with an indid (index ID) value of 0, which means that
the table is a heap. If a primary key or unique constraint is defined on the table, specifying that it is to be supported by
a clustered index, the indid value for the table's row in sysindexes will be 1. Any additional constraints supported by
nonclustered indexes will require that additional rows be added to sysindexes. Each nonclustered index will have a
This document is created with the unregistered version of CHM2PDF Pilot

row with an indid value between 2 and 250. In addition, if there is text or image data in the row, there will be a row in
sysindexes for the text or image data. The name for that row will be the name of the table prefixed by the character t,
and the indid will always be 255. Indid values of 251 through 254 are reserved for future needs.

Data Pages
Data pages are the structures that contain all data that is not text or image. As with all other types of pages in SQL
Server, data pages have a fixed size of 8 KB, or 8192 bytes. They consist of three major components: the page
header, data rows, and the row offset array, as shown in Figure 6-4.

Figure 6-4. The structure of a data page.

Page Header
As you can see in Figure 6-4, the page header occupies the first 96 bytes of each data page (leaving 8096 bytes for
data and row offsets). Table 6-5 shows the information contained in the page header.

Data Rows
Following the page header is the area in which the table's actual data rows are stored. The maximum size of a single
data row is 8060 bytes. A data row can't span multiple pages (except for text or image columns, which can be stored
in their own separate pages). The number of rows stored on a given page will vary depending on the structure of the
table and on the data being stored. A table that has all fixed-length columns will always store the same number of
rows per page; variable-length rows will store as many rows as will fit based on the actual length of the data entered.
Keeping row length shorter allows more rows to fit on a page, thus reducing I/O and improving the cache-hit ratio.

Table 6-5. Information contained in the page header.

Field What It Contains
pageID File number and page number of this page in the database
File number and page number of the next page if this page
nextPage
is in a page chain
File number and page number of the previous page if this
prevPage
page is in a page chain
objID ID of the object to which this page belongs
This document is created with the unregistered version of CHM2PDF Pilot

Log sequence number (LSN) value used for changes and

lsn
updates to this page
slotCnt Total number of slots (rows) used on this page
level Level of this page in an index (always 0 for leaf pages)
indexId Index ID of this page (always 0 for data pages)
freeData Byte offset of the first free space on this page
pminlen Number of bytes in fixed-length portion of rows
freeCnt Number of free bytes on page
reservedCnt Number of bytes reserved by all transactions
Number of bytes reserved by the most recently started
xactreserved
transaction
1 bit per sector for detecting torn page writes (discussed
tornBits
in Chapter 5)
2-byte bitmap that contains additional information about
flagBits
the page

Row Offset Array

The row offset array is a block of 2-byte entries, each of which indicates the offset on the page on which the
corresponding data row begins. Every row has a 2-byte entry in this array. Although these bytes aren't stored in the
row with the data, they do affect the number of rows that will fit on a page.

The row offset array indicates the logical order of rows on a page. For example, if a table has a clustered index, SQL
Server will store the rows in the order of the clustered index key. This doesn't mean that the rows will be physically
stored on the page in the order of the clustered index key. Rather, slot 0 in the offset array will refer to the first row in
the order, slot 1 will refer to the second row, and so forth. As we'll see shortly when we examine an actual page, the
offset of these rows can be anywhere on the page.

There's no internal global row number for every row in a table. SQL Server uses the combination of file number,
page number, and slot number on the page to uniquely identify each row in a table.

Examining Data Pages

You can view the contents of a data page by using the DBCC PAGE statement, which allows you to view the page
header, data rows, and row offset table for any given page in a database. Only a system administrator can use DBCC
PAGE. But because you typically won't need to view the content of a data page, you won't find much about DBCC
PAGE in the SQL Server documentation. Nevertheless, in case you want to use it, here's the syntax:

DBCC PAGE ({dbid | dbname}, filenum, pagenum[, printopt])

The DBCC PAGE command includes the parameters shown in Table 6-6. Figure 6-5 shows sample output from
DBCC PAGE. Note that DBCC TRACEON(3604) instructs SQL Server to return the results to the client instead of
to the error log, as is the default for many of the DBCC commands that deal with internals issues.

Table 6-6. Parameters of the DBCC PAGE command.

Parameter Description
dbid ID of the database containing the page
dbname Name of the database containing the page
filenum File number containing the page
This document is created with the unregistered version of CHM2PDF Pilot

pagenum Page number within the file

Optional print option; takes one of these values:

0: Default; prints the buffer header and page header

1: Prints the buffer header, page header, each row

separately, and the row offset table
printopt
2: Prints the buffer and page headers, the page as a
whole, and the offset table

3: Prints the buffer header, page header, each row

separately, and the row offset table; each row is followed
by each of its column values listed separately
This document is created with the unregistered version of CHM2PDF Pilot
This document is created with the unregistered version of CHM2PDF Pilot

Figure 6-5. Sample output from DBCC PAGE.

As you can see, the output from DBCC PAGE is divided into four main sections: BUFFER, PAGE HEADER,
DATA, and OFFSET TABLE (really the Offset Array). The BUFFER section shows information about the buffer for
the given page. (A buffer in this context is an in-memory structure that manages a page.)

The PAGE HEADER section in Figure 6-5 displays the data for all the header fields on the page. (Table 6-5 shows
the meaning of most of these fields.) The DATA section contains information for each row. For each row, DBCC
PAGE indicates the slot position of the row and the offset of the row on the page. The page data is then divided into
three parts. The left column indicates the byte position within the row where the displayed data occurs. The next four
columns contain the actual data stored on the page, displayed in hexadecimal. The right column contains a character
representation of the data. Only character data is readable in this column, although some of the other data might be
displayed.

The OFFSET TABLE section shows the contents of the row offset array at the end of the page. In the figure, you can
see that this page contains 23 rows, with the first row (indicated by slot 0) beginning at offset 1585 (0x631). The first
row physically stored on the page is actually row 6, with an offset in the row offset array of 96. DBCC PAGE
displays the rows in slot number order, even though, as you can see by the offset of each of the slots, that isn't the
order in which the rows physically exist on the page.

The Structure of Data Rows

A table's data rows have the general structure shown in Figure 6-6. The data for all fixed-length columns is stored
first, followed by the data for all variable-length columns. Table 6-7 shows the information stored in each row.

Status Bits A contains a bitmap indicating properties of the row. The bits have the following meaning:
This document is created with the unregistered version of CHM2PDF Pilot

 Bit 0 Versioning information; in SQL Server 2000, it's always 0.

 Bits 1 through 3 Taken as a 3-bit value, 0 indicates a primary record, 1 indicates a forwarded record, 2
indicates a forwarded stub, 3 indicates an index record, 4 indicates a blob fragment, 5 indicates a ghost index
record, and 6 indicates a ghost data record. (I'll discuss forwarding and ghost records in Chapter 9.)
 Bit 4 Indicates that a NULL bitmap exists; in SQL Server 2000, a NULL bitmap is always present, even if
no NULLs are allowed in any column.
 Bit 5 Indicates that variable-length columns exist in the row.
 Bits 6 and 7 Not used in SQL Server 2000.

Figure 6-6. The structure of data rows.

Table 6-7. Information stored in a table's data rows.

Information Mnemonic Size
Status Bits A TagA 1 byte
Status Bits B (not used in SQL Server
TagB 1 byte
2000)
Fixed-length size Fsize 2 bytes
Fixed-length data Fdata Fsize -4
Number of columns Ncol 2 bytes
NULL bitmap (1 bit for each column
in table; a 1 indicates that the Nullbits Ceiling (Ncol / 8)
corresponding column is NULL)
Number of variable-length columns VarCount 2 bytes
Variable column offset array VarOffset 2 * VarCount
VarOff[VarCount] - (Fsize + 4 +
Variable-length data VarData
Ceiling (Ncol / 8) + 2 * VarCount)

Within each block of fixed-length or variable-length data, the data is stored in the column order in which the table was
created. For example, suppose a table is created with the following statement:
CREATE TABLE Test1
(
Col1 int NOT NULL,
Col2 char(25) NOT NULL,
Col3 varchar(60) NULL,
Col4 money NOT NULL,
Col5 varchar(20) NOT NULL
)
This document is created with the unregistered version of CHM2PDF Pilot

The fixed-length data portion of this row would contain the data for Col1, followed by the data for Col2, followed
by the data for Col4. The variable-length data portion would contain the data for Col3, followed by the data for Col5
. For rows that contain only fixed-length data, the following is true:
 The first hexadecimal digit of the first byte of the data row will be 1, indicating that no variable-length columns
exist. (The first hexadecimal digit is comprised of bits 4 through 7; bits 6 and 7 are always 0, and if no
variable-length columns exist, bit 5 is also 0. Bit 4 is always 1, so the value of the four bits is displayed as 1.)
 The data row ends after the NULL bitmap, which follows the fixed-length data. (That is, the shaded portion
shown in Figure 6-6 won't exist in rows with only fixed-length data.)
 The total length of every data row will be the same.

Column Offset Arrays

A data row that has all fixed-length columns has no variable column count or column offset array. A data row that
has variable-length columns has a column offset array in the data row with a 2-byte entry for each variable-length
column, indicating the position within the row where each column ends. (The terms offset and position aren't exactly
interchangeable. Offset is 0-based, and position is 1-based. A byte at an offset of 7 is in the eighth byte position in
the row.)

Storage of Fixed-Length and Variable-Length

Rows
The two examples that follow illustrate how fixed-length and variable-length data rows are stored. First, the simpler
case of an all fixed-length row:
CREATE TABLE Fixed
(
Col1 char(5) NOT NULL,
Col2 int NOT NULL,
Col3 char(3) NULL,
Col4 char(6) NOT NULL,
Col5 float NOT NULL
)

When this table is created, the following row (or one very much like it) is inserted into the sysindexes system table:
id name indid first minlen
----------- ----- ------ -------------- ------
1797581442 Fixed 0 0xC70000000100 30

And these rows are inserted into the syscolumns system table:
name colid xtype length xoffset
---- ------ ----- ------ -------
Col1 1 175 5 4
Col2 2 56 4 9
Col3 3 175 3 13
Col4 4 175 6 16
Col5 5 62 8 22

For tables containing only fixed-length columns, the minlen value in sysindexes will be equal to the sum of the
column lengths (from syscolumns.length), plus 4 bytes. It won't include the 2 bytes for the number of columns or the
bytes for the NULL bitmap.
This document is created with the unregistered version of CHM2PDF Pilot

To look at a specific data row in this table, you first insert a new row:
INSERT Fixed VALUES ('ABCDE', 123, NULL, 'CCCC', 4567.8)

Figure 6-7 shows this row's actual contents on the data page. To run the DBCC PAGE command, I had to take the
value of first from the syindexes output above (0xC70000000100) and convert it to a file and page address. In
hexadecimal notation, each set of two hexadecimal digits represents a byte. I first had to swap the bytes to get 00 01
00 00 00 C7. The first two groups represent the 2-byte file number, and the last four groups represent the page
number. So the file is 0x0001, which is 1, and the page number is 0x000000C7, which is 199 in decimal.

Figure 6-7. A data row containing all fixed-length columns.

NOTE
The sysindexes table contains three columns that represent page numbers within a database: first, firstIAM, and root
. Each is stored in a byte-swapped format. To convert to a decimal file number and page number, you must first swap
the bytes and then convert the values from hexadecimal to decimal. You could use the Windows calculator to do the
conversion, but a script has been provided on the companion CD to create a stored procedure called
ConvertPageNums, which converts all these columns in the sysindexes table for you.
WARNING
SQL Server does not guarantee that the sysindexes.first column will also indicate the first page of a table. I've found
that first is reliable until you begin to perform deletes and updates on the data in the table.

Reading the output takes a bit of practice. DBCC PAGE displays the data rows in groups of 4 bytes at a time. Within
each group of four, the bytes are listed in reverse order. So the first group of four bytes is byte 3, byte 2, byte 1, and
byte 0. The shaded area in Figure 6-7 has been expanded to show the bytes in the actual byte-number sequence.

The first byte is Status Bits A, and its value (0x10) indicates that only bit 3 is on, so the row has no variable-length
columns. The second byte in the row remains unused. The third and fourth bytes (1e00) indicate the length of the
fixed-length fields, which is also the column offset in which the Ncol value can be found. (The byte-swapped value is
0x001e, which translates to 30.) You can identify the data in the row for each column simply by using the offset value
in the syscolumns table: the data for column Col1 begins at offset 4, the data for column Col2 begins at offset 9, and
so on. As an int, the data in Col2 (7b000000) must be byte-swapped to give us the value 0x0000007b, which is
equivalent to 123 in decimal.

Note that the 3 bytes of data for Col3 are all zeros, representing an actual NULL in the column. Because the row has
This document is created with the unregistered version of CHM2PDF Pilot

no variable-length columns, the row ends 3 bytes after the data for column Col5. The 2 bytes starting right after the
fixed-length data at offset 30 (0500, which is byte-swapped to yield 0x0005) indicate that five columns are in the
row. The last byte is the NULL bitmap. The value of 4 means that only the third bit is on, because in our row, the
third column was indeed a NULL.
WARNING
Fixed-length columns always use the full number of bytes defined for the table, even if the column holds a NULL
value. If you're just upgrading from version 6.5 of SQL Server, this might take you by surprise. I know of at least one
large database at a manufacturing company that expanded to three or four times its size when it was upgraded from
version 6.5. This is because they had fields in many tables that were defined as char(255) and allowed NULLs.
Some tables had 10 or more such columns, and the vast majority of the data was actually NULL. In version 6.5, the
only space that these NULL columns needed was the 2 bytes in the variable column offset array; the actual NULL
data took no space at all. In SQL Server 2000, every one of these columns needed a full 255 bytes of storage and
the database ballooned in size!

Here's the somewhat more complex case of a table with variable-length data. Each row has three varchar columns:
CREATE TABLE Variable
(
Col1 char(3) NOT NULL,
Col2 varchar(250) NOT NULL,
Col3 varchar(5) NULL,
Col4 varchar(20) NOT NULL,
Col5 smallint NULL
)

When you create this table, the following row is inserted into the sysindexes system table:
id name indid first minlen
----------- -------- ------ -------------- ------
1333579789 Variable 0 0xC90000000100 9

And these rows are inserted into the syscolumns system table:
name colid xtype length xoffset
---- ------ ----- ------ -------
Col1 1 175 3 4
Col2 2 167 250 -1
Col3 3 167 5 -2
Col4 4 167 20 -3
Col5 5 52 2 7

Now you insert a row into the table:

INSERT Variable VALUES
('AAA', REPLICATE('X', 250), NULL, 'ABC', 123)

The REPLICATE function is used here to simplify populating a column; this function builds a string of 250 Xs to be
inserted into Col2.

As shown in Figure 6-8, the data for the fixed-length columns is located using the offset value in syscolumns. In this
case, Col1 begins at offset 4, and Col5 begins at offset 7.

To find the variable-length columns, you first locate the column offset array in the row. Right after the 2-byte field
indicating the total number of columns (0x0500) and the NULL bitmap with the value 0x04, a 2-byte field exists with
This document is created with the unregistered version of CHM2PDF Pilot

the value 0x0300 (or 3, decimal) indicating that three variable-length fields exist. Next comes the column offset array.
Three 2-byte values indicate the ending position of each of the three variable-length columns: 0x0e01 is
byte-swapped to 0x010e, so the first variable byte column ends at position 270. The next 2-byte offset is also
0x0e01, so that column has no length and has nothing stored in the variable data area. Unlike with fixed-length fields,
if a variable-length field has a NULL value, it takes no room in the data row. SQL Server distinguishes between a
varchar containing NULL and an empty string by determining whether the bit for the field is 0 or 1 in the NULL
bitmap. The third 2-byte offset is 0x1101, which, when byte-swapped, gives us 0x0111. This means that the row
ends at position 273 (and is a total of 273 bytes in length).

The total storage space needed for a row depends on a number of factors. Variable-length fields add additional
overhead to a row, and their actual size is probably unpredictable. Even for fixed-length fields, the number of bytes of
overhead can change depending on the number of columns in the table. In the earlier example illustrated in Figure 6-1,
I mentioned that 10 bytes of overhead will exist if a row contains all fixed-length columns. For that row, 10 is the
correct number. The size of the NULL bitmap needs to be long enough to store a bit for every column in the row. In
the Figure 6-1 example, the table has 11 columns, so the NULL bitmap needs to be 2 bytes. In the examples
illustrated by Figures 6-7 and 6-8, the table has only 5 columns, so the NULL bitmaps need only a single byte. Don't
forget that the total row overhead also needs to include the 2 bytes for each row in the offset table at the bottom of
the page.

Figure 6-8. A data row with variable-length columns

Page Linkage
SQL Server 2000 doesn't connect the individual data pages of a table in a doubly linked list unless the table has a
clustered index. Pages at each level of an index are linked together, and since the data is considered the leaf level of a
clustered index, SQL Server does maintain the linkage. However, for a heap, there is no such linked list connecting
the pages to each other. The only way that SQL Server determines which pages belong to a table is by inspecting the
IAMs for the table.

If the table has a clustered index, you can use the M_nextPage and M_prevPage values in the page header
information to determine the ordering of pages in the list. Alternatively, you can use the DBCC EXTENTINFO
This document is created with the unregistered version of CHM2PDF Pilot

command to get a list of all the extents that belong to an object. This example uses the Orders table in the Northwind
database:
DBCC EXTENTINFO ('Northwind', 'Orders', 1)

The last argument indicates only extents for index 1, which is the clustered index (and includes the data). Here is the
output:
file_id page_id pg_alloc ext_size obj_id
----------- ----------- ----------- ----------- -----------
1 203 1 1 21575115
1 205 1 1 21575115
1 230 1 1 21575115
1 231 1 1 21575115
1 232 1 1 21575115
1 233 1 1 21575115
1 234 1 1 21575115
1 235 1 1 21575115
1 240 8 8 21575115
1 264 5 8 21575115

index_id pfs_bytes
-------- --------
1 0x60000000?
1 0x60000000?
1 0x60000000?
1 0x60000000?
1 0x60000000?
1 0x60000000?
1 0x60000000?
1 0x60000000?
1 0x40404040?
1 0x40404040?

Notice that the first eight rows indicate an extent size (ext_size) of 1. As discussed in Chapter 5, the first eight pages
of a table are allocated from mixed extents. Only after the table has reached eight pages does SQL Server allocate
uniform extents of eight pages each. The last two rows in the table show this situation, and the page number (page_id)
column gives the page number of the first page of the extent. Note that the last extent (starting on page 264) has used
only five of its pages so far.

Text and Image Data

As mentioned earlier, if a table contains LOB data (text, ntext, or image data), the actual data might not be stored
on the data pages with the rest of the data for a row. In versions prior to SQL Server 2000, SQL Server always
stored a 16-byte pointer in the data row that indicated where the actual data could be found. In SQL Server 2000,
the default behavior is identical to that in SQL Server 7, but you have the option of changing the storage mechanism
by setting a table option to allow LOB data to be stored in the data row itself. So, first I'll tell you about the default
storage of LOB data.

LOB Data Stored Outside the Table

As stated earlier, by default no LOB data is stored in the data row. Instead, the data row contains only a 16-byte
pointer to a page where the data can be found. These pages are 8 KB in size, like any other page in SQL Server, and
individual text, ntext, and image pages aren't limited to holding data for only one occurrence of a text, ntext, or
This document is created with the unregistered version of CHM2PDF Pilot

image column. A text, ntext, or image page can hold data from multiple columns and from multiple rows; the page
can even have a mix of text, ntext, and image data. However, one text or image page can hold only text or image
data from a single table.

The collection of 8-KB pages that make up a LOB field aren't necessarily located next to each other. The pages are
logically organized in a B-tree structure so that operations starting in the middle of the string are very efficient. SQL
Server 2000 can quickly navigate the tree; earlier versions of SQL Server had to scan through the page chain. The
structure of the B-tree differs slightly depending on whether the amount of data is less than or more than 32 KB. (See
Figure 6-9 for the general structure.) I'll discuss B-trees in more detail when I talk about index internals in Chapter 8.

Figure 6-9. A text column pointing to a B-tree that contains the blocks of data.

If the amount of data is less than 32 KB, the text pointer in the data row points to an 84-byte text root structure. This
forms the root node of the B-tree structure. The root node points to the blocks of text, ntext, or image data. While
the data for LOB columns is arranged logically in a B-tree, physically both the root node and the individual blocks of
data are spread throughout LOB pages for the table. They're placed wherever space is available. The size of each
block of data is determined by the size written by an application. Small blocks of data are combined to fill a page. If
the amount of data is less than 64 bytes, it's all stored in the root structure.

If the amount of data for one occurrence of a LOB column exceeds 32 KB, SQL Server starts building intermediate
nodes between the data blocks and the root node. The root structure and the data blocks are interleaved throughout
the text and image pages in the same manner as described earlier. The intermediate nodes, however, are stored in
pages that aren't shared between occurrences of text or image columns. Each page storing intermediate nodes
contains only intermediate nodes for one text or image column in one data row.

LOB Data Stored in the Data Row

If you store all your LOB data outside of your data pages, every time you access that data SQL Server will need to
perform additional page reads. If you need the LOB data frequently, and if some of your data is small enough to fit
within a SQL Server data page, you might notice a performance improvement by allowing some of the LOB data to
be stored in the data row. You can enable a table option called text in row for a particular table by either setting the
option 'ON' or by specifying a maximum number of bytes to be stored in the data row. The following command
enables up to 500 bytes of LOB data to be stored on the data pages of the Employees table in the Northwind
database:
sp_tableoption Employees, 'text in row', 500
This document is created with the unregistered version of CHM2PDF Pilot

Instead of a number, you can specify the option 'ON' (including the quotes), which sets the maximum size to 256
bytes. Note that in the employees table, the only LOB column is notes, which is defined as ntext. This means that
500 bytes allows only 250 characters. Once you enable the text in row option, you never get just the 16-byte pointer
for the LOB data in the row, as is the case when the option is not on. If the data in the LOB field is more than the
specified maximum, the row will hold the root structure containing pointers to the separate chunks of LOB data. The
minimum size of a root structure is 24 bytes, and the possible range of values that text in row can be set to is 24 to
7000 bytes.

To disable the text in row option, you can set the value to either 'OFF' or 0. To determine whether a table has the
text in row property enabled, you can use the OBJECTPROPERTY function, as shown here:
SELECT OBJECTPROPERTY (object_id('Employees'), 'TableTextInRowLimit')

This function returns the maximum number of bytes allowed for storing LOBs in a data row. If a 0 is returned, the
text in row option is disabled.

Let's create a table very similar to the one we created to look at row structures, but we'll change the varchar(250)
column to the text datatype. We'll use almost the same insert statement to insert one row into the table.
CREATE TABLE HasText
(
Col1 char(3) NOT NULL,
Col2 text NOT NULL,
Col3 varchar(5) NULL,
Col4 varchar(20) NOT NULL,
Col5 smallint NULL
)

INSERT HasText VALUES

('AAA', REPLICATE('X', 250), NULL, 'ABC', 123)

Now let's find the necessary information for this table from the sysindexes table and note the results:
SELECT indid, first, name FROM sysindexes
WHERE id = object_id('HasText')

Results:

indid first name

------ -------------- --------------
0 0xBF0000000100 HasText
0xBD0000000100 tHasText

Even though there are no indexes on this table, there are two rows in sysindexes. Sysindexes keeps track of all
objects that take up space, and since a LOB field takes space separate from the table, it has its own row in
sysindexes. The index ID for LOB data is always 255, and the "name" of the structure is the name of the table
prefixed by the letter t. The first page of the table is page 0xBF, which is 191 decimal, and the LOB data is at 0xBD,
which is 189 decimal. Figure 6-10 shows the output from running DBCC PAGE. The row structure is very similar to
the row structure in Figure 6-x, except for the text field itself. Bytes 20 to 35 are the 16-byte text pointer, and you
can see the value bd (or 0xBD) at offset 31. This is the pointer to the page where the text data starts, and as we saw
from the row in sysindexes, that is indeed the location of the LOB data for this table.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 6-10. A row containing a text pointer.

Now let's enable text data in the row, for up to 500 bytes:
EXEC sp_tableoption HasText, 'text in row', 500

Enabling this option does not force the text data to be moved into the row. We have to update the text value to
actually force the data movement:
UPDATE HasText
SET col2 = REPLICATE('Z', 250)

Figure 6-11 shows the output from running DBCC PAGE after the text has been moved into the row. Note that the
row is practically identical to the row containing varchar data, as shown in Figure 6-8.

Figure 6-11. A row containing text data.

Finally, let's change the maximum size of in-row text data to only 50 bytes. This will force the text data to be moved
off the page, and the root of the text structure will be stored in the row. As long as the text in row option is not OFF
(or 0), SQL Server will never store the simple 16-byte LOB pointer in the row. It will store either the LOB data itself,
if it fits, or the root structure for the LOB data B-tree.
EXEC sp_tableoption HasText, 'text in row', 50

Figure 6-12 shows the output from running DBCC PAGE after the text pointer root has been stored on the page. A
root structure is at least 24 bytes long (which is why 24 is the minimum size for setting the text-in-row limit.) Again,
you can see the page number 0xBD at offset 40, indicating the location of the actual text data. Other information in the
root includes:
 Bytes 0 through 1: The type of column; 1 indicates a LOB root
 Byte 2: Level in the B-tree
 Byte 3: Unused
This document is created with the unregistered version of CHM2PDF Pilot

 Bytes 4 through 7: A value used by optimistic concurrency control for cursors that increases every time a
LOB is updated
 Bytes 8 through 11: A random value used by DBCC CHECKTABLE that remains unchanged during the
lifetime of each LOB
 Bytes 12 through 23 and each succeeding group of 12 bytes in the column: Links to LOB data on a separate
page

Figure 6-12. A row containing a text pointer root.

As indicated above, when you first enable text in row, no data movement occurs until text data is actually updated.
The same is true if the limit is increased—that is, even if the new limit is large enough to accommodate LOB data that
had been stored outside the row, the LOB data will not be moved onto the row automatically. You must update the
actual LOB data first.

However, if the text in row option is turned OFF or the limit is reduced, all LOB data must adjust to comply with
the new specifications. If the option is set to OFF, all LOB data is moved off the rows and replaced by a 16-byte text
pointer. If the option limit is decreased, all LOB data larger than the new limit is moved off the rows and replaced by
a LOB root pointer. You should be aware that SQL Server will log all movement of LOB data, which means that
reducing the limit of or turning OFF the text in row option can be a very time-consuming operation for a large table.

Another point to keep in mind is that just because the amount of LOB data is less than the limit, that does not always
mean the data will be stored in the row. You're still limited to a maximum row size of 8060 bytes for a single row on a
data page. In addition, if a variable-length column needs to grow, it might push LOB data off of the page so as not to
exceed the 8060-byte limit. Growth of variable-length columns always has priority over storing LOB data in the row
because the non-LOB data must be stored on the data page. If no variable-length char fields need to grow during an
update operation, SQL Server will check for growth of in-row LOB data, in column offset order. If one LOB needs
to grow, others might be pushed off the row.

sql_variant Datatype
The new sql_variant data type provides support for columns that contain any or all of SQL Server's base datatypes
except LOBs, rowversion (timestamp), and the types that can't be defined for a column in a table, namely cursor
and table. For instance, a column can contain a smallint value in some rows, a float value in others, and a char value
in the remainder.

This feature was designed to support what appears to be semistructured data in products sitting above SQL Server.
This semistructured data exists in conceptual tables that have a fixed number of columns of known datatypes and one
or more optional columns whose type might not be known in advance. An example is e-mail messages in Microsoft
Outlook and Microsoft Exchange. With the sql_variant type, you can pivot a conceptual table into a real, more
compact table with sets of property-value pairs. Here is a graphical example. The conceptual table shown in Table
6-8 has three rows of data. The fixed columns are the ones that exist in every row. Each row can also have values for
one or more of the three different properties, which have different datatypes.

Table 6-8. Conceptual table with an arbitrary number of columns and datatypes.
fixed-columns property-1 property-2 property-3
row-1 XXXXXX value-11 value-13
This document is created with the unregistered version of CHM2PDF Pilot

row-2 YYYYYY value-22

row-3 ZZZZZZ value-31 value-32

This can be pivoted into Table 6-9, where the fixed columns are repeated for each different property that appears
with those columns. The column called value can be represented by sql_variant data and be of a different datatype
for each different property.

Table 6-9. Semi-structured data stored using the SQL_VARIANT datatype.

fixed-columns property value
XXXXXX property-1 value-11
XXXXXX property-3 value-13
YYYYYY property-2 value-22
ZZZZZZ property-1 value-31
ZZZZZZ property-2 value-32

Internally, columns of type sql_variant are always considered variable-length. Their storage structure depends on the
type of data, but the first byte of every sql_variant field always indicates the actual datatype being used in that row.

I'll create a simple table with a sql_variant column and insert a few rows into it so we can observe the structure of
the sql_variant storage.
USE pubs
GO
CREATE TABLE var (a int, b sql_variant)
GO
INSERT INTO var VALUES (1, 3)
INSERT INTO var VALUES (2, 3000000000)
INSERT INTO var VALUES (3, 'abc')
INSERT INTO var VALUES (4, current_timestamp)

SQL Server decides what datatype to use in each row based on the data supplied. For example, the 3 in the first
INSERT is assumed to be an integer. In the second INSERT, the 3000000000 is larger than the biggest possible
integer, so SQL Server assumes a decimal with a precision of 10 and a scale of 0. (It could have used a bigint, but
that would need more storage space.) We can now find the first page of the table and use DBCC PAGE to see its
contents:
SELECT first FROM sysindexes WHERE name = 'var'
-- (I got a value of 0x8E0000000100 for first, so that can be
-- converted to a value of 142 decimal.)
GO
DBCC TRACEON (3604)
GO
DBCC PAGE (5, 1, 142, 1)

Figure 6-13 shows the contents of the four rows. I won't go into the details of every single byte because most are the
same as what we've already examined.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 6-13. Rows containing sql_variant data.

The difference between the three rows starts at bytes 13-14, which indicate the position where the first
variable-length column ends. Since there is only one variable-length column, this is also the length of the row. The
sql_variant data begins at byte 15. Byte 15 is the code for the datatype. You can find the codes in the xtype column
of the systypes system table. I've reproduced the relevant part of that table here:
xtype name
----- ----------------
34 image
35 text
36 uniqueidentifier
48 tinyint
52 smallint
56 int
58 smalldatetime
59 real
60 money
61 datetime
62 float
98 sql_variant
99 ntext
104 bit
106 decimal
108 numeric
122 smallmoney
127 bigint
165 varbinary
167 varchar
167 tid
167 id
This document is created with the unregistered version of CHM2PDF Pilot

173 binary
175 char
175 empid
189 timestamp
231 sysname
231 nvarchar
239 nchar

In our table, we have the datatypes 38 hex (which is 56 decimal, which is int), 6C hex (which is 108 decimal, which
is numeric), A7 hex (which is 167 decimal, which is varchar) and 3D hex (which is 61 decimal, which is datetime).
Following the byte for datatype is a byte representing the version of the sql_variant format, and that is always 1 in
SQL Server 2000. Following the version, there can be one of the following four sets of bytes:
 For numeric and decimal: 1 byte for the precision and 1 byte for the scale
 For strings: 2 bytes for the maximum length and 4 bytes for the collation ID
 For binary and varbinary: 2 bytes for the maximum length
 For all other types: no extra bytes

These bytes are then followed by the actual data in the sql_variant column.

There are many other issues related to working with sql_variant data and comparing data of different types. I'll cover
these when we look at data retrieval and manipulation in Chapter 7.
This document is created with the unregistered version of CHM2PDF Pilot

34

Constraints
Constraints provide a powerful yet easy way to enforce the data integrity in your database. As described in Chapter
2, data integrity comes in three forms:
 Entity integrity ensures that a table has a primary key. In SQL Server 2000, you can guarantee entity integrity
by defining PRIMARY KEY or UNIQUE constraints or by building unique indexes. Alternatively, you can
write a trigger to enforce entity integrity, but this is usually far less efficient.
 Domain integrity ensures that data values meet certain criteria. In SQL Server 2000, domain integrity can be
guaranteed in several ways. Choosing appropriate datatypes can ensure that a data value meets certain
conditions—for example, that the data represents a valid date. Other approaches include defining CHECK
constraints or FOREIGN KEY constraints or writing a trigger. You might also consider DEFAULT
constraints as an aspect of enforcing domain integrity.
 Referential integrity enforces relationships between two tables, a referenced table, and a referencing table.
SQL Server 2000 allows you to define FOREIGN KEY constraints to enforce referential integrity, and you
can also write triggers for enforcement. It's crucial to note that there are always two sides to referential
integrity enforcement. If data is updated or deleted from the referenced table, referential integrity ensures that
any data in the referencing table that refers to the changed or deleted data is handled in some way. On the
other side, if data is updated or inserted into the referencing table, referential integrity ensures that the new
data matches a value in the referenced table.

In this section, I'll discuss the use of constraints for enforcing your data integrity requirements. Constraints are also
called declarative data integrity because they are part of the actual table definition. This is in contrast to
programmatic data integrity enforcement, which uses stored procedures or triggers. Here are the five types of
constraints:
 PRIMARY KEY
 UNIQUE
 FOREIGN KEY
 CHECK
 DEFAULT

You might also sometimes see the IDENTITY property and the nullability of a column described as constraints. I
typically don't consider these attributes to be constraints; instead, I think of them as properties of a column, for two
reasons. First, as we'll see, each constraint has its own row in the sysobjects system table, but IDENTITY and
nullability information is not stored in sysobjects, only in syscolumns. This makes me think that these properties are
more like datatypes, which are also stored in syscolumns. Second, when you use the special command SELECT
INTO, a new table can be created that is a copy of an existing table. All column names and datatypes are copied, as
well as IDENTITY information and column nullability. However, constraints are not copied to the new table. This
makes me think that IDENTITY and nullability are more a part of the actual table structure than constraints are.

PRIMARY KEY and UNIQUE Constraints

A central tenet of the relational model is that every row in a table is in some way unique and can be distinguished in
some way from every other row in the table. You could use the combination of all columns in a table as this unique
identifier, but the identifier is usually at most the combination of a handful of columns, and often it's just one column:
the primary key. Although some tables might have multiple unique identifiers, each table can have only one primary
key. For example, perhaps the employee table maintains both an Emp_ID column and an SSN (social security
number) column, both of which can be considered unique. Such column pairs are often referred to as alternate keys
or candidate keys, although both terms are design terms and aren't used by the ANSI SQL standard or by SQL
Server. In practice, one of the two columns is logically promoted to primary key using the PRIMARY KEY
This document is created with the unregistered version of CHM2PDF Pilot

constraint, and the other is usually declared by a UNIQUE constraint.Although neither the ANSI SQL standard nor
SQL Server require it, it's good practice to declare a PRIMARY KEY constraint on every table. Furthermore, you
must designate a primary key for a table that will be published for transaction-based replication.

Internally, PRIMARY KEY and UNIQUE constraints are handled almost identically, so I'll discuss them together
here. Declaring a PRIMARY KEY or UNIQUE constraint simply results in a unique index being created on the
specified column or columns, and this index enforces the column's uniqueness in the same way that a unique index
created manually on a column would. The query optimizer makes decisions based on the presence of the unique index
rather than on the fact that a column was declared as a primary key. How the index got there in the first place is
irrelevant to the optimizer.

Nullability
All columns that are part of a primary key must be declared (either explicitly or implicitly) as NOT NULL. Columns
that are part of a UNIQUE constraint can be declared to allow NULL. However, for the purposes of unique indexes,
all NULLs are considered equal. So if the unique index is on a single column, only one NULL value can be stored
(another good reason to try to avoid NULL whenever possible). If the unique index is on a composite key, one of the
columns can have many NULLs as long as the value in the other column is unique. For example, if the constraint
contains two int columns, exactly one row of each of these combinations will be allowed:
NULL NULL
0 NULL
NULL 0
1 NULL
NULL 1

This behavior is questionable: NULL represents an unknown, but using it this way clearly implies that NULL is equal
to NULL. As you'll recall, I recommend that you avoid using NULLs, especially in key columns.

Index Attributes
You can explicitly specify the index attributes CLUSTERED or NONCLUSTERED when you declare a constraint.
If you don't, the index for a UNIQUE constraint will be nonclustered and the index for a PRIMARY KEY constraint
will be clustered (unless CLUSTERED has already been explicitly stated for a unique index, because only one
clustered index can exist per table). You can specify the index FILLFACTOR attribute if a PRIMARY KEY or
UNIQUE constraint is added to an existing table using the ALTER TABLE command. FILLFACTOR doesn't make
sense in a CREATE TABLE statement because the table has no existing data, and FILLFACTOR on an index affects
how full pages are only when the index is initially created. FILLFACTOR isn't maintained when data is added.

Choosing Keys
Try to keep the key lengths as compact as possible. Columns that are the primary key or that are unique are most
likely to be joined and frequently queried. Compact key lengths allow more index entries to fit on a given 8-KB page,
thereby reducing I/O, increasing cache hits, and speeding up character matching. Clustered index keys are used as
bookmarks in all your nonclustered indexes, so a long clustered key will increase the size and decrease the I/O
efficiency of all your indexes. So if your primary key has a clustered index, you've got plenty of good reasons to keep
it short. When no naturally efficient compact key exists, it's often useful to manufacture a surrogate key using the
IDENTITY property on an int column. If int doesn't provide enough range, a good second choice is a numeric
column with the required precision and with scale 0. Alternatively, you can consider a bigint for an identity column
You might use this surrogate as the primary key, use it for most join and retrieval operations, and declare a UNIQUE
constraint on the natural but inefficient columns that provide the logical unique identifier in your data. Or you might
dispense with creating the UNIQUE constraint altogether if you don't need to have SQL Server enforce the
uniqueness. Indexes slow performance of data modification statements because the index as well as the data must be
maintained.
This document is created with the unregistered version of CHM2PDF Pilot

Although it's permissible to do so, don't create a PRIMARY KEY constraint on a column of type float or real.
Because these are approximate datatypes, the uniqueness of such columns is also approximate, and the results can
sometimes be unexpected.

Removing PRIMARY KEY or UNIQUE Constraints

You can't directly drop a unique index created as a result of a PRIMARY KEY or UNIQUE constraint by using the
DROP INDEX statement. Instead, you must drop the constraint by using ALTER TABLE DROP CONSTRAINT
(or you must drop the table itself). This feature was designed so that a constraint can't be compromised accidentally
by someone who doesn't realize that the index is being used to enforce the constraint.

There is no way to temporarily disable a PRIMARY KEY or UNIQUE constraint. If disabling is required, use
ALTER TABLE DROP CONSTRAINT and then later restore the constraint by using ALTER TABLE ADD
CONSTRAINT. If the index you use to enforce uniqueness is clustered, when you add the constraint to an existing
table, the entire table and all nonclustered indexes will be internally rebuilt to establish the cluster order. This can be a
time-consuming task, and it requires about 1.2 times the existing table space as a temporary work area in the
database would require (2.2 times total, counting the permanent space needed) so that the operation can be rolled
back if necessary. However, you can rebuild an index that is enforcing a constraint by using the DROP_EXISTING
option of the CREATE INDEX statement. This drops and re-creates the index behind the scenes as a single
operation. I'll discuss the CREATE INDEX command in detail, including the DROP_EXISTING option, in Chapter
8.

Creating PRIMARY KEY and UNIQUE Constraints

Typically, you declare PRIMARY KEY and UNIQUE constraints when you create the table (CREATE TABLE).
However, you can add or drop both by subsequently using the ALTER TABLE command. To simplify matters, you
can declare a PRIMARY KEY or UNIQUE constraint that includes only a single column on the same line that you
define that column in the CREATE TABLE statement. Or you can declare the constraint as a separate line in your
CREATE TABLE statement. I'll show you examples of both methods shortly. The approach you use is largely a
matter of personal preference unless your constraint involves multiple columns. Constraints defined along with a
column definition can refer only to that column.

Following are examples of four ways to declare a PRIMARY KEY constraint on a single column. They all cause a
unique, clustered index to be created. Note the (abridged) output of the sp_helpconstraint procedure for
each—especially the constraint name.

EXAMPLE 1
CREATE TABLE customer
(
cust_id int IDENTITY NOT NULL PRIMARY KEY,
cust_name varchar(30) NOT NULL
)
GO

EXEC sp_helpconstraint customer

GO

>>>>

Object Name
-----------
customer

constraint_type constraint_name
----------------------- ------------------------------
This document is created with the unregistered version of CHM2PDF Pilot

PRIMARY KEY (clustered) PK__customer__68E79C55

EXAMPLE 2
CREATE TABLE customer
(
cust_id int IDENTITY NOT NULL
CONSTRAINT cust_pk PRIMARY KEY,
cust_name varchar(30) NOT NULL
)
GO

EXEC sp_helpconstraint customer

GO

>>>>

Object Name
-----------
customer

constraint_type constraint_name
----------------------- ---------------
PRIMARY KEY (clustered) cust_pk

No foreign keys reference this table.

EXAMPLE 3
CREATE TABLE customer
(
cust_id int IDENTITY NOT NULL,
cust_name varchar(30) NOT NULL,
PRIMARY KEY (cust_id)
)
GO

EXEC sp_helpconstraint customer

GO

>>>>

Object Name
-----------
customer

constraint_type constraint_name
--------------- ---------------
PRIMARY KEY (clustered) PK__customer__59063A47

No foreign keys reference this table.

EXAMPLE 4
CREATE TABLE customer
(
cust_id int IDENTITY NOT NULL,
cust_name varchar(30) NOT NULL,
CONSTRAINT customer_PK PRIMARY KEY (cust_id)
)
GO
This document is created with the unregistered version of CHM2PDF Pilot

EXEC sp_helpconstraint customer

GO

>>>>

Object Name
-----------
customer

constraint_type constraint_name
--------------- ---------------
PRIMARY KEY (clustered) customer_PK

No foreign keys reference this table.

In Examples 1 and 3, I did not provide an explicit name for the constraint, so SQL Server came up with the names.
The names, PK__customer__68E79C55 and PK__customer__59063A47, seem cryptic, but there is some method
to this apparent madness. All types of single-column constraints use this same naming scheme, which I'll discuss later
in the chapter, and multi-column constraints use a similar scheme. Whether you choose a more intuitive name, such as
cust_pk in Example 2 or customer_PK in Example 4, or the less intuitive (but information-packed),
system-generated name is up to you. Here's an example of creating a multi-column, UNIQUE constraint on the
combination of multiple columns. (The primary key case is essentially identical.)
CREATE TABLE customer_location
(
cust_id int NOT NULL,
cust_location_number int NOT NULL,
CONSTRAINT customer_location_unique UNIQUE
(cust_id, cust_location_number)
)
GO

EXEC sp_helpconstraint customer_location

GO

>>>
Object Name
-----------------
customer_location

constraint_type constraint_name constraint_keys

--------------- ------------------------ ---------------
UNIQUE customer_location_unique cust_id,
(non-clustered) cust_location_number

No foreign keys reference this table.

As noted earlier, a unique index is created to enforce either a PRIMARY KEY or a UNIQUE constraint. The name
of the index is the same as the constraint name, whether the name was explicitly defined or system-generated. The
index used to enforce the CUSTOMER_LOCATION_UNIQUE constraint in the above example is also named
customer_location_unique. The index used to enforce the column-level, PRIMARY KEY constraint of the
customer table in Example 1 is named PK__customer__68E79C55, which is the system-generated name of the
constraint. You can use the sp_helpindex stored procedure to see information for all indexes of a given table. For
example:
EXEC sp_helpindex customer

>>>
index_name index_description index_keys
This document is created with the unregistered version of CHM2PDF Pilot

------------------------------ ------------------ ----------

PK__customer__68E79C55 clustered, unique, cust_id
primary key
located on default

FOREIGN KEY Constraints

One of the fundamental concepts of the relational model is the logical relationships between tables. In most
databases, certain relationships must exist (that is, the data must have referential integrity) or else the data will be
logically corrupt.

SQL Server automatically enforces referential integrity through the use of FOREIGN KEY constraints. This feature is
sometimes referred to as declarative referential integrity, or DRI, to distinguish it from other features, such as triggers,
that you can also use to enforce the existence of the relationships.

Referential Actions
The full ANSI SQL-92 standard contains the notion of the referential action, sometimes (incompletely) referred to
as a cascading delete. SQL Server 7 doesn't provide this feature as part of FOREIGN KEY constraints, but this
notion warrants some discussion here because the capability exists via triggers.

The idea behind referential actions is this: sometimes, instead of just preventing an update of data that would violate a
foreign key reference, you might be able to perform an additional, compensating action that will still enable the
constraint to be honored. For example, if you were to delete a customer table that has references to orders, you can
have SQL Server automatically delete all those related order records (that is, cascade the delete to orders), in which
case the constraint won't be violated and the customer table can be deleted. This feature is intended for both
UPDATE and DELETE statements, and four possible actions are defined: NO ACTION, CASCADE, SET
DEFAULT, and SET NULL.
 NO ACTION The delete is prevented. This default mode, per the ANSI standard, occurs if no other action
is specified. NO ACTION is often referred to as RESTRICT, but this usage is slightly incorrect in terms of
how ANSI defines RESTRICT and NO ACTION. ANSI uses RESTRICT in DDL statements such as
DROP TABLE, and it uses NO ACTION for FOREIGN KEY constraints. (The difference is subtle and
unimportant. It's common to refer to the FOREIGN KEY constraint as having an action of RESTRICT.)
 CASCADE A delete of all matching rows in the referencing table occurs.
 SET DEFAULT The delete is performed, and all foreign key values in the referencing table are set to a
default value.
 SET NULL The delete is performed, and all foreign key values in the referencing table are set to NULL.

SQL Server 2000 allows you to specify either NO ACTION or CASCADE when you define your foreign key
constraints. I'll give you some examples shortly. If you want to implement either SET DEFAULT or SET NULL, you
can use a trigger. Implementing the SET NULL action is very straightforward, as you'll see in Chapter 12, which
discusses triggers in detail. One reason it's so straightforward is that while a FOREIGN KEY requires any value in the
referencing table to match a value in the referenced table, NULLs are not considered to be a value, and having a
NULL in the referencing table doesn't break any relationships. Implementing the SET DEFAULT action is just a little
more involved. Suppose you want to remove a row from the referenced table and set the foreign key value in all
referencing rows to -9999. If you want to maintain referential integrity, you have to have a "dummy" row in the
referenced table with the primary key value of -9999.

Keep in mind that enforcing a foreign key implies that SQL Server will check data in both the referenced and the
referencing tables. The referential actions just discussed apply only to actions that are taken on the referenced table.
On the referencing table, the only action allowed is to not allow an update or an insert if the relationship will be broken
as a result.
NOTE
This document is created with the unregistered version of CHM2PDF Pilot

Because a constraint is checked before a trigger fires, you can't have both a FOREIGN KEY constraint to enforce
the relationship when a new key is inserted into the referencing table and a trigger that performs an operation such as
SET NULL on the referenced table. If you do have both, the constraint will fail and the statement will be aborted
before the trigger to cascade the delete fires. If you want to allow the SET NULL action, you have to write two
triggers. You have to have a trigger for delete and update on the referenced table to perform the SET NULL action
and a trigger for insert and update on the referencing table that disallows new data that would violate the referential
integrity.

If you do decide to enforce your referential integrity with triggers, you might still want to declare the foreign key
relationship largely for readability so that the relationship between the tables is clear. You can then use the
NOCHECK option of ALTER TABLE to disable the constraint, and then the trigger will fire.

This note is just referring to AFTER triggers. SQL Server 2000 provides an additional type of trigger called an
INSTEAD OF trigger, which I'll talk about in Chapter 12.

Here's a simple way to declare a primary key/foreign key relationship:

CREATE TABLE customer
(
cust_id int NOT NULL IDENTITY PRIMARY KEY,
cust_name varchar(50) NOT NULL
)

CREATE TABLE orders

(
order_id int NOT NULL IDENTITY PRIMARY KEY,
cust_id int NOT NULL REFERENCES customer(cust_id)
)

Since the default referential action is NO ACTION, the orders table could also have been written in this way:
CREATE TABLE orders
(
order_id int NOT NULL IDENTITY PRIMARY KEY,
cust_id int NOT NULL REFERENCES customer(cust_id)
ON UPDATE NO ACTION ON DELETE NO ACTION
)

The orders table contains the column cust_id, which references the primary key of the customer table. An order (
order_id) must not exist unless it relates to an existing customer (cust_id). With this definition, you can't delete a row
from the customer table if a row that references it currently exists in the orders table, and you can't modify the
cust_id column in a way that would destroy the relationship.

Note that a referential action can be specified for both DELETE and UPDATE operations on the referenced table,
and the two operations can have different actions. For example, you can choose to CASCADE any updates to the
cust_id in customer but not allow (NO ACTION) any deletes of referenced rows in customer.

Here's an example of adding a FOREIGN KEY constraint to the orders table that does just that: it cascades any
updates to the cust_id in customer but does not allow any deletes of referenced rows in customer:
CREATE TABLE orders
(
order_id int NOT NULL IDENTITY PRIMARY KEY,
cust_id int NOT NULL REFERENCES customer(cust_id)
ON UPDATE CASCADE ON DELETE NO ACTION
)
This document is created with the unregistered version of CHM2PDF Pilot

The previous examples show the syntax for a single-column constraint—both the primary key and foreign key are
single columns. This syntax uses the keyword REFERENCES, and the term "foreign key" is implied but not explicitly
stated. The name of the FOREIGN KEY constraint is generated internally, following the same general form described
earlier for PRIMARY KEY and UNIQUE constraints. Here's a portion of the output of sp_helpconstraint for both
the customer and orders tables. (The tables were created in the pubs sample database.)
EXEC sp_helpconstraint customer
>>>

Object Name
------------
customer

constraint_type constraint_name constraint_keys

---------------- ---------------------- ---------------
PRIMARY KEY PK__customer__07F6335A cust_id
(clustered)

Table is referenced by
----------------------------------------------
pubs.dbo.orders: FK__orders__cust_id__0AD2A005

EXEC sp_helpconstraint orders

>>>

Object Name
-----------
orders

constraint_type constraint_name delete_ update_

action action
--------------- ----------------------------- --------- ------
FOREIGN KEY FK__orders__cust_id__0AD2A005 no action no action
PRIMARY KEY PK__orders__09DE7BCC
(clustered)

constraint_keys

---------------
cust_id REFERENCES pubs.dbo.customer(cust_id)
order_id

No foreign keys reference this table.

Constraints defined inline with the column declaration can reference only a single column. This is also true for
PRIMARY KEY and UNIQUE constraints. You must declare a multi-column constraint as a separate table element.
The following example shows a multi-column FOREIGN KEY constraint:
CREATE TABLE customer
(
cust_id int NOT NULL,
location_num smallint NULL,
cust_name varchar(50) NOT NULL,
CONSTRAINT CUSTOMER_UNQ UNIQUE CLUSTERED (location_num, cust_id)
)
This document is created with the unregistered version of CHM2PDF Pilot

CREATE TABLE orders

(
order_id int NOT NULL IDENTITY CONSTRAINT ORDER_PK
PRIMARY KEY NONCLUSTERED,
cust_num int NOT NULL,
cust_loc smallint NULL,
CONSTRAINT FK_ORDER_CUSTOMER FOREIGN KEY (cust_loc, cust_num)
REFERENCES customer (location_num, cust_id)
)
GO

EXEC sp_helpconstraint customer

EXEC sp_helpconstraint orders
GO

>>>
Object Name
-----------
customer

constraint_type constraint_name constraint_keys

------------------ ---------------- --------------------
UNIQUE (clustered) CUSTOMER_UNQ location_num, cust_id

Table is referenced by
---------------------------
pubs.dbo.orders: FK_ORDER_CUSTOMER

Object Name
------------
orders

constraint_type constraint_name constraint_keys

---------------- ---------------- ------------------
FOREIGN KEY FK_ORDER_CUSTOMER cust_loc, cust_num
REFERENCES pubs.dbo.customer
(location_num, cust_id)

PRIMARY KEY ORDER_PK order_id

(non-clustered)
No foreign keys reference this table.

The preceding example also shows the following variations on how you can create constraints:
 You can use a FOREIGN KEY constraint to reference a UNIQUE constraint (an alternate key) instead of a
PRIMARY KEY constraint. (Note, however, that referencing a PRIMARY KEY is much more typical and is
generally better practice.)
 You don't have to use identical column names in tables involved in a foreign key reference, but doing so is
often good practice. The cust_id and location_num column names are defined in the customer table. The
orders table, which references the customer table, uses the names cust_num and cust_loc.
 The datatypes of the related columns must be identical, except for nullability and variable-length attributes.
(For example, a column of char(10) NOT NULL can reference one of varchar(10) NULL, but it can't
reference a column of char(12) NOT NULL. A column of type smallint can't reference a column of type int
.) Notice in the preceding example that cust_id and cust_num are both int NOT NULL and that
location_num and cust_loc are both smallint NULL.

Unlike a PRIMARY KEY or UNIQUE constraint, an index isn't automatically built for the column or columns
declared as FOREIGN KEY. However, in many cases, you'll want to build indexes on these columns because they're
often used for joining to other tables. To enforce foreign key relationships, SQL Server must add additional steps to
This document is created with the unregistered version of CHM2PDF Pilot

the execution plan of every insert, delete, and update (if the update affects columns that are part of the relationship)
that affects either the table referencing another table or the table being referenced. The execution plan, determined by
the SQL Server optimizer, is simply the collection of steps that carries out the operation. (In Chapter 16, you'll see the
actual execution plan by using the SHOWPLAN options.)

If no FOREIGN KEY constraints exist, a statement specifying the update of a single row of the orders table might
have an execution plan like the following:
1. Find a qualifying order record using a clustered index.
2. Update the order record.

When a FOREIGN KEY constraint exists on the orders table, the same operation has more steps in the execution
plan:
1. Check for the existence of a related record in the customer table (based on the updated order record) using
a clustered index.
2. If no related record is found, raise an exception and terminate the operation.
3. Find a qualifying order record using a clustered index.
4. Update the order record.

The execution plan is more complex if the orders table has many FOREIGN KEY constraints declared. Internally, a
simple update or insert operation might no longer be possible. Any such operation requires checking many other
tables for matching entries. Because a seemingly simple operation might require checking as many as 253 other tables
(see the next paragraph) and possibly creating multiple worktables, the operation might be much more complicated
than it looks and much slower than expected.

A table can have a maximum of 253 FOREIGN KEY references. This limit is derived from the internal limit of 256
tables in a single query. In practice, an operation on a table with 253 or fewer FOREIGN KEY constraints might still
fail with an error because of the 256-table query limit if worktables are required for the operation.

A database designed for excellent performance doesn't reach anything close to this limit of 253 FOREIGN KEY
references. For best performance results, use FOREIGN KEY constraints judiciously. Some sites unnecessarily use
too many FOREIGN KEY constraints because the constraints they declare are logically redundant. Take the
following example. The orders table declares a FOREIGN KEY constraint to both the master_customer and
customer_location tables:
CREATE TABLE master_customer
(
cust_id int NOT NULL IDENTITY PRIMARY KEY,
cust_name varchar(50) NOT NULL
)

CREATE TABLE customer_location

(
cust_id int NOT NULL,
cust_loc smallint NOT NULL,
CONSTRAINT PK_CUSTOMER_LOCATION PRIMARY KEY (cust_id,cust_loc),
CONSTRAINT FK_CUSTOMER_LOCATION FOREIGN KEY (cust_id)
REFERENCES master_customer (cust_id)
)

CREATE TABLE orders

(
order_id int NOT NULL IDENTITY PRIMARY KEY,
cust_id int NOT NULL,
cust_loc smallint NOT NULL,
CONSTRAINT FK_ORDER_MASTER_CUST FOREIGN KEY (cust_id)
REFERENCES master_customer (cust_id),
CONSTRAINT FK_ORDER_CUST_LOC FOREIGN KEY (cust_id, cust_loc)
REFERENCES customer_location (cust_id, cust_loc)
)
This document is created with the unregistered version of CHM2PDF Pilot

Although logically the relationship between the orders and master_customer tables exists, the relationship is
redundant to and subsumed by the fact that orders is related to customer_location, which has its own FOREIGN
KEY constraint to master_customer. Declaring a foreign key for master_customer adds unnecessary overhead
without adding any further integrity protection.
NOTE
In the case just described, declaring a foreign key improves the readability of the table definition, but you can achieve
the same result by simply adding comments to the CREATE TABLE command. It's perfectly legal to add a comment
practically anywhere—even in the middle of a CREATE TABLE statement. A more subtle way to achieve this result
is to declare the constraint so that it appears in sp_helpconstraint and in the system catalogs but then disable the
constraint by using the ALTER TABLE NOCHECK option. Because the constraint will then be unenforced, an
additional table isn't added to the execution plan.

The CREATE TABLE statement shown in the following example for the orders table omits the redundant foreign key
and, for illustrative purposes, includes a comment. Despite the lack of a FOREIGN KEY constraint in the master_
customer table, you still can't insert a cust_id that doesn't exist in the master_customer table because the reference
to the customer_location table will prevent it.
CREATE TABLE orders
(
order_id int NOT NULL IDENTITY PRIMARY KEY,
cust_id int NOT NULL,
cust_loc smallint NOT NULL,
—- Implied Foreign Key Reference of:
—- (cust_id) REFERENCES master_customer (cust_id)
CONSTRAINT FK_ORDER_CUST_LOC FOREIGN KEY (cust_id, cust_loc)
REFERENCES customer_location (cust_id, cust_loc)
)

Note that the table on which the foreign key is declared (the referencing table, which in this example is orders) isn't
the only table that requires additional execution steps. When being updated, the table being referenced (in this case
customer_location) must also have additional steps in its execution plan to ensure that an update of the columns
being referenced won't break a relationship and create an orphan entry in the orders table. Without making any
changes directly to the customer_location table, you can see a significant decrease in update or delete performance
because of foreign key references added to other tables.

Practical Considerations for FOREIGN KEY Constraints

When using constraints, you should consider triggers, performance, and indexing. Let's take a look at the
ramifications of each.

Triggers I won't discuss triggers in detail until Chapter 12, but for now, you should simply note that constraints are
enforced before an AFTER triggered action is performed. If the constraint is violated, the statement will abort before
the trigger fires.
NOTE
The owner of a table isn't allowed to declare a foreign key reference to another table unless the owner of the other
table has granted REFERENCES permission to the first table owner. Even if the owner of the first table is allowed to
select from the table to be referenced, that owner must have REFERENCES permission. This prevents another user
from changing the performance of operations on your table without your knowledge or consent. You can grant any
user REFERENCES permission even if you don't also grant SELECT permission, and vice-versa. The only exception
is that the DBO, or any user who is a member of the db_owner role, has full default permissions on all objects in the
database.
This document is created with the unregistered version of CHM2PDF Pilot

Performance When deciding on the use of foreign key relationships, you must weigh the protection provided against
the corresponding performance overhead. Be careful not to add constraints that form logically redundant relationships.
Excessive use of FOREIGN KEY constraints can severely degrade the performance of seemingly simple operations.

Indexing The columns specified in FOREIGN KEY constraints are often strong candidates for index creation. You
should build the index with the same key order used in the PRIMARY KEY or UNIQUE constraint of the table that
it references so that joins can be performed efficiently. Also be aware that a foreign key is often a subset of the table's
primary key. In the customer_location table used in the preceding two examples, cust_id is part of the primary key
as well as a foreign key in its own right. Given that cust_id is part of a primary key, it's already part of an index. In
this example, cust_id is the lead column of the index, and building a separate index on it alone probably isn't
warranted. However, if cust_id is not the lead column of the index B-tree, it might make sense to build an index on it.

Constraint-Checking Solutions
Sometimes two tables reference one another, which creates a bootstrap problem. Suppose Table1 has a foreign key
reference to Table2, but Table2 has a foreign key reference to Table1. Even before either table contains any data,
you'll be prevented from inserting a row into Table1 because the reference to Table2 will fail. Similarly, you can't
insert a row into Table2 because the reference to Table1 will fail.

ANSI SQL has a solution: deferred constraints, in which you can instruct the system to postpone constraint
checking until the entire transaction is committed. Using this elegant remedy puts both INSERT statements into a
single transaction that results in the two tables having correct references by the time COMMIT occurs. Unfortunately,
no mainstream product currently provides the deferred option for constraints. The deferred option is part of the
complete SQL-92 specification, which no product has yet fully implemented.

SQL Server 2000 provides immediate constraint checking; it has no deferred option. SQL Server offers three
options for dealing with constraint checking: it allows you to add constraints after adding data, it lets you temporarily
disable checking of foreign key references, and it allows you to use the bcp (bulk copy) program or BULK INSERT
command to initially load data and avoid checking FOREIGN KEY constraints. (You can override this default option
with bcp or the BULK INSERT command and force FOREIGN KEY constraints to be validated.) To add
constraints after adding data, don't create constraints via the CREATE TABLE command. After adding the initial
data, you can add constraints by using the ALTER TABLE command.

With the second option, the table owner can temporarily disable checking of foreign key references by using the
ALTER TABLE table NOCHECK CONSTRAINT statement. Once data exists, you can reestablish the FOREIGN
KEY constraint by using ALTER TABLE table CHECK CONSTRAINT. Note that when an existing constraint is
reenabled using this method, SQL Server doesn't automatically check to see that all rows still satisfy the constraint. To
do this, you can simply issue a dummy update by setting a column to itself for all rows, determining whether any
constraint violations are raised, and then fixing them. (For example, you can issue UPDATE orders SET cust_id =
cust_id.)

Finally you can use the bcp program or the BULK INSERT command to initially load data. The BULK INSERT
command and the bcp program don't check any FOREIGN KEY constraints by default. You can use the
CHECK_CONSTRAINTS option to override this behavior. BULK INSERT and bcp are faster than regular
INSERT commands because they usually bypass normal integrity checks and most logging.

When you use ALTER TABLE to add (instead of reenable) a new FOREIGN KEY constraint for a table in which
data already exists, the existing data is checked by default. If constraint violations occur, the constraint isn't added.
With large tables, such a check can be quite time-consuming. You do have an alternative—you can add a FOREIGN
KEY constraint and omit the check. To do this, you specify the WITH NOCHECK option with ALTER TABLE. All
subsequent operations will be checked, but existing data won't be checked. As in the case of reenabling a constraint,
you can then perform a dummy update to flag any violations in the existing data. If you use this option, you should do
the dummy update as soon as possible to ensure that all the data is clean. Otherwise, your users might see constraint
error messages when they perform update operations on the preexisting data even if they haven't changed any values.
This document is created with the unregistered version of CHM2PDF Pilot

Restrictions on Dropping Tables

If you're dropping tables, you must drop all the referencing tables or drop the referencing FOREIGN KEY
constraints before dropping the referenced table. For example, in the preceding example's orders,
customer_location, and master_customer tables, the following sequence of DROP statements fails because a table
being dropped is referenced by a table that still exists—that is, customer_location can't be dropped because the
orders table references it, and orders isn't dropped until later:
DROP TABLE customer_location
DROP TABLE master_customer
DROP TABLE orders

Changing the sequence to the following works fine because orders is dropped first:
DROP TABLE orders
DROP TABLE customer_location
DROP TABLE master_customer

When two tables reference each other, you must first drop the constraints or set them to NOCHECK (both
operations use ALTER TABLE) before the tables can be dropped. Similarly, a table that's being referenced can't be
part of a TRUNCATE TABLE command. You must drop or disable the constraint or else simply drop and rebuild
the table.

Self-Referencing Tables
A table can be self-referencing—that is, the foreign key can reference one or more columns in the same table. The
following example shows an employee table in which a column for managers references another employee entry:
CREATE TABLE employee
(
emp_id int NOT NULL PRIMARY KEY,
emp_name varchar(30) NOT NULL,
mgr_id int NOT NULL REFERENCES employee(emp_id)
)

The employee table is a perfectly reasonable table. It illustrates most of the issues I've discussed. However, in this
case, a single INSERT command that satisfies the reference is legal. For example, if the CEO of the company has an
emp_id of 1 and is also his own manager, the following INSERT will be allowed and can be a useful way to insert the
first row in a self-referencing table:
INSERT employee VALUES (1, 'Chris Smith', 1)

Although SQL Server doesn't currently provide a deferred option for constraints, self-referencing tables add a twist
that sometimes makes SQL Server use deferred operations internally. Consider the case of a nonqualified DELETE
statement that deletes many rows in the table. After all rows are ultimately deleted, you can assume that no constraint
violation will occur. However, violations might occur during the DELETE operation because some of the remaining
referencing rows might be orphaned before they are actually deleted. SQL Server handles such interim violations
automatically and without any user intervention. As long as the self-referencing constraints are valid at the end of the
data modification statement, no errors are raised during processing.

To gracefully handle these interim violations, however, additional processing and worktables are required to hold the
This document is created with the unregistered version of CHM2PDF Pilot

work-in-progress. This adds substantial overhead and can also limit the actual number of foreign keys that can be
used. An UPDATE statement can also cause an interim violation. For example, if all employee numbers are to be
changed by multiplying each by 1000, the following UPDATE statement would require worktables to avoid the
possibility of raising an error on an interim violation:
UPDATE employee SET emp_id=emp_id * 1000, mgr_id=mgr_id * 1000

The additional worktables and the processing needed to handle the worktables are made part of the execution plan.
Therefore, if the optimizer sees that a data modification statement could cause an interim violation, the additional
temporary worktables will be created even if no such interim violations ever actually occur. These extra steps are
needed only in the following situations:
 A table is self-referencing (it has a FOREIGN KEY constraint that refers back to itself).
 A single data modification statement (UPDATE, DELETE, or INSERT based on a SELECT) is performed
and can affect more than one row. (The optimizer can't determine a priori, based on the WHERE clause and
unique indexes, whether more than one row could be affected.) Multiple data modification statements within
the transaction don't apply—this condition must be a single statement that affects multiple rows.
 Both the referencing and referenced columns are affected (which is always the case for DELETE and
INSERT operations, but might or might not be the case for UPDATE).

If a data modification statement in your application meets the preceding criteria, you can be sure that SQL Server is
automatically using a limited and special-purpose form of deferred constraints to protect against interim violations.

CHECK Constraints
Enforcing domain integrity (that is, ensuring that only entries of expected types, values, or ranges can exist for a
given column) is also important. SQL Server provides two ways to enforce domain integrity: CHECK constraints and
rules. CHECK constraints allow you to define an expression for a table that must not evaluate to FALSE for a data
modification statement to succeed. (Note that I didn't say that the constraint must evaluate to TRUE. The constraint
will allow the row if it evaluates to TRUE or to unknown. The constraint evaluates to unknown when NULL values
are present, and this introduces three-value logic. The issues of NULLs and three-value logic are discussed in depth
in Chapter 7.) SQL Server provides a similar mechanism to CHECK constraints, called rules, which are provided
basically for backward compatibility reasons. Rules perform almost the same function as CHECK constraints, but
they use different syntax and have fewer capabilities. Rules have existed in SQL Server since its initial release in 1989,
well before CHECK constraints, which are part of the ANSI SQL-92 standard and were added in version 6 in
1995.

CHECK constraints make a table's definition more readable by including the domain checks in the DDL. Rules have a
potential advantage in that they can be defined once and then bound to multiple columns and tables (using
sp_bindrule each time), while you must respecify a CHECK constraint for each column and table. But the extra
binding step can also be a hassle, so this capability for rules is beneficial only if a rule will be used in many places.
Although performance between the two approaches is identical, CHECK constraints are generally preferred over
rules because they're a direct part of the table's DDL, they're ANSI-standard SQL, they provide a few more
capabilities than rules (such as the ability to reference other columns in the same row or to call a system function), and
perhaps most important, they're more likely than rules to be further enhanced in future releases of SQL Server. For
these reasons, I'm not going to talk about rules any more.

CHECK constraints add additional steps to the execution plan to ensure that the expression doesn't evaluate to
FALSE (which would result in the operation being aborted). Although steps are added to the execution plan for data
modifications, these are typically much less expensive than the extra steps discussed earlier for FOREIGN KEY
constraints. For foreign key checking, another table must be searched, requiring additional I/O. CHECK constraints
deal only with some logical expression for the specific row already being operated on, so no additional I/O is
required. Because additional processing cycles are used to evaluate the expressions, the system requires more CPU
use. But if there's plenty of CPU power to spare, the effect might well be negligible. You can watch this by using
This document is created with the unregistered version of CHM2PDF Pilot

Performance Monitor.

As with other constraint types, you can declare CHECK constraints on a single column or on multiple columns. You
must declare a CHECK constraint that refers to more than one column as a separate element in the CREATE
TABLE statement. Only single-column CHECK constraints can be defined right along with the column definition, and
only one CHECK constraint can be defined with the column. All other CHECK constraints have to be defined as
separate elements. However, keep in mind that one constraint can have multiple logical expressions that can be
AND'ed or OR'ed together.

Some CHECK constraint features have often been underutilized by SQL Server database designers, including the
ability to reference other columns in the same row, use system and niladic functions (which are evaluated at runtime),
use combinations of expressions combined with AND or OR, and use CASE expressions. The following example
shows a table with multiple CHECK constraints (as well as a PRIMARY KEY constraint and a FOREIGN KEY
constraint) and showcases some of these features:
CREATE TABLE employee
(
emp_id int NOT NULL PRIMARY KEY
CHECK (emp_id BETWEEN 0 AND 1000),

emp_name varchar(30) NOT NULL CONSTRAINT no_nums

CHECK (emp_name NOT LIKE '%[0-9]%'),

mgr_id int NOT NULL REFERENCES employee(emp_id),

entered_date datetime NULL CHECK (entered_date >=

CURRENT_TIMESTAMP),

entered_by int CHECK (entered_by IS NOT NULL),

CONSTRAINT valid_entered_by CHECK
(entered_by = SUSER_ID(NULL) AND
entered_by <> emp_id),

CONSTRAINT valid_mgr CHECK (mgr_id <> emp_id OR emp_id=1),

CONSTRAINT end_of_month CHECK (DATEPART(DAY, GETDATE()) < 28)

)
GO

EXEC sp_helpconstraint employee

GO
>>>>

Object Name
---------------
employee

constraint_type constraint_name constraint_keys

--------------- -------------------------------- ---------------
CHECK on column CK__employee__emp_id__2C3393D0 ([emp_id] >= 0
emp_id [emp_id] <=
1000)

CHECK on column CK__employee__entered_by__300424B4 (((not(

entered_by [entered_by]
is null))))

CHECK on column CK__employee__entered_date__2F10007B ([entered_date]

entered_date >= getdate())

CHECK Table end_of_month (datepart(day,

This document is created with the unregistered version of CHM2PDF Pilot

Level getdate()) <

28)

FOREIGN KEY FK__employee__mgr_id__2E1BDC42 mgr_id

CHECK on column no_nums ([emp_name] not

emp_name like '%[0-9]%')

PRIMARY KEY PK__employee__2B3F6F97 emp_id

(clustered)

CHECK Table Level valid_entered_by ([entered_by] =

suser_id(null)
and
[entered_by] <>
[emp_id])

CHECK Table Level valid_mgr ([mgr_id] <>

[emp_id] or
[emp_id] = 1)

Table is referenced by
--------------------------------
pubs.dbo.employee: FK__employee__mgr_id__2E1BDC42

This example illustrates the following points:

 CHECK constraints can be expressed at the column level with abbreviated syntax (leaving naming to SQL
Server), such as the check on entered_date; at the column level with an explicit name, such as the
NO_NUMS constraint on emp_name; or as a table-level constraint, such as the VALID_MGR constraint.
 CHECK constraints can use regular expressions—for example, NO_NUMS ensures that a digit can never be
entered as a character in a person's name.
 Expressions can be AND'ed and OR'ed together to represent more complex situations—for example,
VALID_MGR. However, SQL Server won't check for logical correctness at the time a constraint is added to
the table. Suppose you want to restrict the values in a department_id column to either negative numbers or
values greater than 100. (Perhaps numbers between 0 and 100 are reserved.) You can add this column and
constraint to the table using ALTER TABLE.
ALTER TABLE employee
add department_no int CHECK
(department_no < 0 AND department_no > 100)

However, once the preceding command executes successfully and the new column and constraint are added
to the table, you can't add another row to the table. I accidentally typed AND in the ALTER TABLE
statement instead of OR, but that isn't considered an illegal CHECK constraint. Each time I try to insert a row,
SQL Server will try to validate that the value in the department_no column is both less than 0 and more than
100! Any value for department_no that doesn't meet that requirement will cause the insert to fail. It could
take a long time to come up with a value to satisfy this constraint. In this case, I'd have to drop the constraint
and add a new one with the correct definition before I could insert any rows.
 Table-level CHECK constraints can refer to more than one column in the same row. For example,
VALID_MGR means that no employee can be his own boss, except employee number 1, who is assumed to
be the CEO. SQL Server currently has no provision that allows you to check a value from another row or
from a different table.
 You can make a CHECK constraint prevent NULL values—for example, CHECK (entered_by IS NOT
NULL). Generally, you simply declare the column NOT NULL.
 A NULL column might make the expression logically unknown. For example, a NULL value for
entered_date gives CHECK entered_date >= CURRENT_TIMESTAMP an unknown value. This doesn't
reject the row, however. The constraint rejects the row only if the expression is clearly false, even if it isn't
This document is created with the unregistered version of CHM2PDF Pilot

necessarily true.
 You can use system functions, such as GETDATE, APP_NAME, DATALENGTH, and SUSER_ID, as well
as niladic functions, such as SYSTEM_USER, CURRENT_TIMESTAMP, and USER, in CHECK
constraints. This subtle feature is powerful and can be useful, for example, to ensure that a user can change
only records that she has entered by comparing entered_by to the user's system ID, as generated by
SUSER_ID (or by comparing emp_name to SYSTEM_USER). Note that niladic functions such as
CURRENT_TIMESTAMP are provided for ANSI SQL conformance and simply map to an underlying SQL
Server function, in this case GETDATE. So while the DDL to create the constraint on entered_date uses
CURRENT_TIMESTAMP, sp_helpconstraint shows it as GETDATE, which is the underlying function.
Either expression is valid, and they are equivalent in the CHECK constraint. The VALID_ENTERED_BY
constraint ensures that the entered_by column can be set only to the currently connected user's ID, and it
ensures that users can't update their own records.
 A constraint defined as a separate table element can call a system function without referencing a column in the
table. In the example preceding this list, the END_OF_MONTH CHECK constraint calls two date functions,
DATEPART and GETDATE, to ensure that updates can't be made after day 27 of the month (which is when
the business's payroll is assumed to be processed). The constraint never references a column in the table.
Similarly, a CHECK constraint might call the APP_NAME function to ensure that updates can be made only
from an application of a certain name, instead of from an ad hoc tool such as SQL Query Analyzer.

As with FOREIGN KEY constraints, you can add or drop CHECK constraints by using ALTER TABLE. When
adding a constraint, by default the existing data is checked for compliance; you can override this default by using the
WITH NOCHECK syntax. You can later do a dummy update to check for any violations. The table or database
owner can also temporarily disable CHECK constraints by using NOCHECK in the ALTER TABLE statement.

Default Constraints
A default allows you to specify a constant value, NULL, or the run-time value of a system function if no known value
exists or if the column is missing in an INSERT statement. Although you could argue that a default isn't truly a
constraint (because a default doesn't enforce anything), you can create defaults in a CREATE TABLE statement using
the CONSTRAINT keyword; therefore, I'll refer to defaults as constraints. Defaults add little overhead, and you can
use them liberally without too much concern about performance degradation.

SQL Server provides two ways of creating defaults. First, you can create a default (CREATE DEFAULT) and then
bind the default to a column (sp_bindefault). This has been the case since the original SQL Server release. Second,
you can use default constraints. These were introduced in 1995 with SQL Server 6.0 as part of the CREATE
TABLE and ALTER TABLE statements, and they're based on the ANSI SQL standard (which includes such niceties
as being able to use system functions). Using defaults is pretty intuitive. The type of default you use is a matter of
preference; both perform the same function internally. Future enhancements are likely to be made to the ANSI-style
implementation. Having the default within the table DDL seems a cleaner approach. I recommend that you use
defaults within CREATE TABLE and ALTER TABLE rather than within CREATE DEFAULT, so I'll focus on that
style here.

Here's the example from the previous CHECK constraint discussion, now modified to include several defaults:
CREATE TABLE employee
(
emp_id int NOT NULL PRIMARY KEY DEFAULT 1000
CHECK (emp_id BETWEEN 0 AND 1000),

emp_name varchar(30) NULL DEFAULT NULL CONSTRAINT no_nums

CHECK (emp_name NOT LIKE '%[0-9]%'),

mgr_id int NOT NULL DEFAULT (1) REFERENCES

employee(emp_id),

entered_date datetime NOT NULL CHECK (entered_date >=

This document is created with the unregistered version of CHM2PDF Pilot

CONVERT(char(10), CURRENT_TIMESTAMP, 102))

CONSTRAINT def_today DEFAULT
(CONVERT(char(10), GETDATE(), 102)),

entered_by int NOT NULL DEFAULT SUSER_ID()

CHECK (entered_by IS NOT NULL),

CONSTRAINT valid_entered_by CHECK (entered_by=SUSER_ID() AND

entered_by <> emp_id),

CONSTRAINT valid_mgr CHECK (mgr_id <> emp_id OR emp_id=1),

CONSTRAINT end_of_month CHECK (DATEPART(DAY, GETDATE()) < 28)

)
GO

EXEC sp_helpconstraint employee

GO

>>>

Object Name
---------------
employee

constraint_type constraint_name constraint_keys

--------------- -------------------------------- ---------------
CHECK on column CK__employee__emp_id__2C3393D0 ([emp_id] >= 0
emp_id [emp_id] <=
1000)

CHECK on column CK__employee__entered_by__300424B4 (((not(

entered_by [entered_by]
is null))))

DEFAULT on column def_today convert(

entered_date char(10),
getdate(),
102))

DEFAULT on column DF__employee__emp_id__35BCFE0A (1000)

emp_id

DEFAULT on column DF__employee__emp_name__37A5467C (null)

emp_name

DEFAULT on column DF__employee__entered_by__3D5E1FD2 (suser_id())

entered_by

DEFAULT on column DF__employee__mgr_id__398D8EEE (1)

mgr_id

CHECK on column CK__employee__entered_date__2F10007B ([entered_date]

entered_date >= getdate())

CHECK Table Level end_of_month (datepart(day,

getdate()) <
28)

FOREIGN KEY FK__employee__mgr_id__2E1BDC42 mgr_id

CHECK on column no_nums ([emp_name]

This document is created with the unregistered version of CHM2PDF Pilot

emp_name not like

'%[0-9]%')

PRIMARY KEY PK__employee__2B3F6F97 emp_id

(clustered)

CHECK Table Level valid_entered_by (

[entered_by] =
suser_id(null)
and
[entered_by]
<> [emp_id])

CHECK Table Level valid_mgr ([mgr_id] <>

[emp_id] or
[emp_id] = 1)

Table is referenced by
--------------------------------
pubs.dbo.employee: FK__employee__mgr_id__2E1BDC42

The preceding code demonstrates the following about defaults:

 A default constraint is always a single-column constraint because it pertains only to one column and can be
defined only along with the column definition; you cannot define a default constraint as a separate table
element. You can use the abbreviated syntax that omits the keyword CONSTRAINT and the specified name,
letting SQL Server generate the name, or you can specify the name by using the more verbose
CONSTRAINT name DEFAULT syntax.
 A default value can clash with a CHECK constraint. This problem appears only at runtime, not when you
create the table or when you add the default using ALTER TABLE. For example, a column with a default of
0 and a CHECK constraint that states that the value must be greater than 0 can never insert or update the
default value.
 Although you can assign a default to a column that has a PRIMARY KEY or a UNIQUE constraint, it
doesn't make much sense to do so. Such columns must have unique values, so only one row could exist with
the default value in that column. The preceding example sets a DEFAULT on a primary key column for
illustration, but in general, this practice is unwise.
 You can write a constant value within parentheses, as in DEFAULT (1), or without them, as in DEFAULT 1.
A character or date constant must be enclosed in either single or double quotation marks.
 One tricky concept is knowing when a NULL is inserted into a column as opposed to a default value. A
column declared NOT NULL with a default defined uses the default only under one of the following
conditions:
o The INSERT statement specifies its column list and omits the column with the default.
o The INSERT statement specifies the keyword DEFAULT in the values list (whether the column is
explicitly specified as part of the column list or implicitly specified in the values list and the column list is
omitted, meaning "All columns in the order in which they were created"). If the values list explicitly
specifies NULL, an error is raised and the statement fails; the default value isn't used. If the INSERT
statement omits the column entirely, the default is used and no error occurs. (This behavior is in
accordance with ANSI SQL.) The keyword DEFAULT can be used in the values list, and this is the only
way the default value will be used if a NOT NULL column is specified in the column list of an INSERT
statement (either, as in the following example, by omitting the column list—which means all columns—or
by explicitly including the NOT NULL column in the columns list).
INSERT EMPLOYEE VALUES (1, 'The Big Guy', 1, DEFAULT, DEFAULT)

Table 6-10 summarizes the behavior of INSERT statements based on whether a column is declared NULL or NOT
NULL and whether it has a default specified. It shows the result for the column for three cases:
 Omitting the column entirely (no entry)
This document is created with the unregistered version of CHM2PDF Pilot

 Having the INSERT statement use NULL in the values list

 Specifying the column and using DEFAULT in the values list

Table 6-10. INSERT behavior with defaults.

No Entry Enter NULL Enter DEFAULT
No Default Default No Default Default No Default Default
NULL NULL default NULL NULL NULL default
NOT NULL error default error error error default

Declaring a default on a column that has the IDENTITY property doesn't make sense, and SQL Server will raise an
error if you try it. The IDENTITY property acts as a default for the column. But the DEFAULT keyword cannot be
used as a placeholder for an identity column in the values list of an INSERT statement. You can use a special form of
INSERT statement if a table has a default value for every column (an identity column does meet this criteria) or allows
NULL. The following statement uses the DEFAULT VALUES clause instead of a column list and values list:
INSERT employee DEFAULT VALUES

TIP
You can generate some test data by putting the IDENTITY property on a primary key column and declaring default
values for all other columns and then repeatedly issuing an INSERT statement of this form within a Transact-SQL
loop.

More About Constraints

This section offers some advice on working with constraints. I'll look at constraint names and system catalog entries,
the status field, constraint failures, multiple-row data modifications, and integrity checks.

Constraint Names and System Catalog Entries

Earlier in this chapter, you learned about the cryptic-looking constraint names that SQL Server generates. Now I'll
explain the naming. Consider again the following simple CREATE TABLE statement:
CREATE TABLE customer
(
cust_id int IDENTITY NOT NULL PRIMARY KEY,
cust_name varchar(30) NOT NULL
)

The constraint produced from this simple statement bears the nonintuitive name PK__customer__68E79C55. All
types of single-column constraints use this naming scheme, which I'll explain shortly. The advantage of explicitly
naming your constraint rather than using the system-generated name is greater clarity. The constraint name is used in
the error message for any constraint violation, so creating a name such as CUSTOMER_PK probably makes more
sense to users than a name such as PK__customer__cust_i__0677FF3C. You should choose your own constraint
names if such error messages are visible to your users. The first two characters (PK) show the constraint type—PK
for PRIMARY KEY, UQ for UNIQUE, FK for FOREIGN KEY, and DF for DEFAULT. Next are two
underscore characters, which are used as a separator. (You might be tempted to use one underscore to conserve
characters and to avoid having to truncate as much. However, it's common to use a single underscore in a table name
or a column name, both of which appear in the constraint name. Using two underscore characters distinguishes the
kind of a name it is and where the separation occurs.)

Next comes the table name (customer), which is limited to 116 characters for a PRIMARY KEY constraint and
slightly fewer characters for all other constraint names. For all constraints other than PRIMARY KEY, there are then
This document is created with the unregistered version of CHM2PDF Pilot

two more underscore characters for separation followed by the next sequence of characters, which is the column
name. The column name is truncated to five characters if necessary. If the column name has fewer than five characters
in it, the length of the table name portion can be slightly longer.

And finally, the hexadecimal representation of the object ID for the constraint (68E79C55) comes after another
separator. (This value is used as the id column of the sysobjects system table and the constid column of the
sysconstraints system table.) Object names are limited to 128 characters in SQL Server 2000, so the total length of
all portions of the constraint name must also be less than or equal to 128.

Note the queries and output using this value that are shown in Figure 6-14.

Figure 6-14. Querying the system tables using the hexadecimal ID value.
TIP
The hexadecimal value 0x68E79C55 is equal to the decimal value 1760009301, which is the value of constid in
sysconstraints and of id in sysobjects.

These sample queries of system tables show the following:

 A constraint is an object. A constraint has an entry in the sysobjects table in the xtype column of C, D, F,
PK, or UQ for CHECK, DEFAULT, FOREIGN KEY, PRIMARY KEY, and UNIQUE, respectively.
 Sysconstraints relates to sysobjects. The sysconstraints table is really just a view of the sysobjects system
table. The constid column in the view is the object ID of the constraint, and the id column of sysconstraints
is the object ID of the base table on which the constraint is declared.
 Colid values in sysconstraints indicate the column the constraint applies to. If the constraint is a
column-level CHECK, FOREIGN KEY, or DEFAULT constraint, sysconstraints.colid has the column ID
of the column. This colid in sysconstraints is related to the colid of syscolumns for the base table
represented by id. A table-level constraint or any PRIMARY KEY/UNIQUE constraint (even if column
level) always has 0 in this column.
TIP
To see the names and order of the columns in a PRIMARY KEY or UNIQUE constraint, you can query the
sysindexes and syscolumns tables for the index being used to enforce the constraint. The name of the constraint and
that of the index enforcing the constraint are the same, whether the name was user-specified or system-generated. The
columns in the index key are somewhat cryptically encoded in the keys1 and keys2 fields of sysindexes. The easiest
way to decode these values is to simply use the sp_helpindex system stored procedure; alternatively, you can use the
code of that procedure as a template if you need to decode them in your own procedure.

Decoding the status Field

The status field of the sysconstraints view is a pseudo-bit-mask field packed with information. We could also call it
a bitmap, because each bit has a particular meaning. If you know how to crack this column, you can essentially write
This document is created with the unregistered version of CHM2PDF Pilot

your own sp_helpconstraint-like procedure. Note that the documentation is incomplete regarding the values of this
column. One way to start decoding this column is to look at the definition of the sysconstraints view using the
sp_helptext system procedure.

The lowest four bits, obtained by AND'ing status with 0xF (status & 0xF), contain the constraint type. A value of 1
is PRIMARY KEY, 2 is UNIQUE, 3 is FOREIGN KEY, 4 is CHECK, and 5 is DEFAULT. The fifth bit is on (
status & 0x10 <> 0) when the constraint is a nonkey constraint on a single column. The sixth bit is on (status &
0x20 <> 0) when the constraint is on multiple columns or is a PRIMARY KEY or UNIQUE constraint.
NOTE
Some of the documentation classifies constraints as either table-level or column-level. This implies that any constraint
defined on the line with a column is a column-level constraint and any constraint defined as a separate line in the table
or added to the table with the ALTER TABLE command is a table-level constraint. However, this distinction does
not hold true when you look at sysconstraints. Although it is further documented that the fifth bit is for a column-level
constraint, you can see for yourself that this bit is on for any single column constraint except PRIMARY KEY and
UNIQUE and that the sixth bit, which is documented as indicating a table-level constraint, is on for all multi-column
constraints, as well as PRIMARY KEY and UNIQUE constraints.

Some of the higher bits are used for internal status purposes, such as noting whether a nonclustered index is being
rebuilt, and for other internal states. Table 6-11 shows some of the other bit-mask values you might be interested in:

Table 6-11. Bitmap values in the status column of sysconstraints.

Bitmap Value Description
The constraint is a "column-level" constraint, which means
16 that it's a single column constraint and isn't enforcing entity
integrity
The constraint is a "table-level" constraint, which means
32 that it's either a multi-column constraint, a PRIMARY
KEY, or a UNIQUE constraint
512 The constraint is enforced by a clustered index.
1024 The constraint is enforced by a nonclustered index.
16384 The constraint has been disabled.
32767 The constraint has been enabled.
131072 SQL Server has generated a name for the constraint

Using this information, and not worrying about the higher bits used for internal status, you could use the following
query to show constraint information for the employee table:
SELECT
OBJECT_NAME(constid) 'Constraint Name',
constid 'Constraint ID',
CASE (status & 0xF)
WHEN 1 THEN 'Primary Key'
WHEN 2 THEN 'Unique'
WHEN 3 THEN 'Foreign Key'
WHEN 4 THEN 'Check'
WHEN 5 THEN 'Default'
ELSE 'Undefined'
END 'Constraint Type',
CASE (status & 0x30)
WHEN 0x10 THEN 'Column'
WHEN 0x20 THEN 'Table'
ELSE 'NA'
END 'Level'
FROM sysconstraints
WHERE id=OBJECT_ID('employee')
This document is created with the unregistered version of CHM2PDF Pilot

>>>

Constraint Name Constraint Constraint Level

ID Type
-------------------------------------- ----------- ------------ ------
PK__employee__49C3F6B7 1237579447 Primary Key Table
DF__employee__emp_id__4AB81AF0 1253579504 Default Column
CK__employee__emp_id__4BAC3F29 1269579561 Check Column
DF__employee__emp_name__4CA06362 1285579618 Default Column
no_nums 1301579675 Check Column
DF__employee__mgr_id__4E88ABD4 1317579732 Default Column
FK__employee__mgr_id__4F7CD00D 1333579789 Foreign Key Column
CK__employee__entered_date__5070F446 1349579846 Check Column
def_today 1365579903 Default Column
DF__employee__entered_by__52593CB8 1381579960 Default Column
CK__employee__entered_by__534D60F1 1397580017 Check Column
valid_entered_by 1413580074 Check Table
valid_mgr 1429580131 Check Table
end_of_month 1445580188 Check Table

Constraint Failures in Transactions and Multiple-Row Data

Modifications
Many bugs occur in application code because the developers don't understand how failure of a constraint affects a
multiple-statement transaction declared by the user. The biggest misconception is that any error, such as a constraint
failure, automatically aborts and rolls back the entire transaction. On the contrary, after an error is raised, it's up to the
transaction to either proceed and ultimately commit or to roll back. This feature provides the developer with the
flexibility to decide how to handle errors. (The semantics are also in accordance with the ANSI SQL-92 standard for
COMMIT behavior).

Following is an example of a simple transaction that tries to insert three rows of data. The second row contains a
duplicate key and violates the PRIMARY KEY constraint. Some developers believe that this example wouldn't insert
any rows because of the error that occurs in one of the statements; they think that this error will cause the entire
transaction to be aborted. However, this doesn't happen—instead, the statement inserts two rows and then commits
that change. Although the second INSERT fails, the third INSERT is processed because no error checking has
occurred between the statements, and then the transaction does a COMMIT. Because no instructions were provided
to take some other action after the error other than to proceed, SQL Server does just that. It adds the first and third
INSERT statements to the table and ignores the second statement.
IF EXISTS (SELECT * FROM sysobjects WHERE name='show_error' AND
type='U')
DROP TABLE show_error
GO

CREATE TABLE show_error

(
col1 smallint NOT NULL PRIMARY KEY,
col2 smallint NOT NULL
)
GO

BEGIN TRANSACTION

INSERT show_error VALUES (1, 1)

INSERT show_error VALUES (1, 2)
INSERT show_error VALUES (2, 2)

COMMIT TRANSACTION
This document is created with the unregistered version of CHM2PDF Pilot

GO

SELECT * FROM show_error

GO

Server: Msg 2627, Level 14, State 1, Line 1

Violation of PRIMARY KEY constraint 'PK__show_error__6EF57B66'.
Cannot insert duplicate key in object 'show_error'.
The statement has been terminated.

col1 col2
---- ----
1 1
2 2

Here's a modified version of the transaction. This example does some simple error checking using the system function
@@ERROR and rolls back the transaction if any statement results in an error. In this example, no rows are inserted
because the transaction is rolled back.
IF EXISTS (SELECT * FROM sysobjects WHERE name='show_error'
AND type='U')
DROP TABLE show_error
GO

CREATE TABLE show_error

(
col1 smallint NOT NULL PRIMARY KEY,
col2 smallint NOT NULL
)
GO

BEGIN TRANSACTION
INSERT show_error VALUES (1, 1)
IF @@ERROR <> 0 GOTO TRAN_ABORT
INSERT show_error VALUES (1, 2)
if @@ERROR <> 0 GOTO TRAN_ABORT
INSERT show_error VALUES (2, 2)
if @@ERROR <> 0 GOTO TRAN_ABORT
COMMIT TRANSACTION
GOTO FINISH

TRAN_ABORT:
ROLLBACK TRANSACTION

FINISH:
GO

SELECT * FROM show_error

GO

Server: Msg 2627, Level 14, State 1, Line 1

Violation of PRIMARY KEY constraint 'PK__show_error__70DDC3D8'.
Cannot insert duplicate key in object 'show_error'.
The statement has been terminated.

col1 col2
---- ----
(0 row(s) affected)

Because many developers have handled transaction errors incorrectly and because it can be tedious to add an error
check after every command, SQL Server includes a SET statement that aborts a transaction if it encounters any error
This document is created with the unregistered version of CHM2PDF Pilot

during the transaction. (Transact-SQL has no WHENEVER statement, although such a feature would be useful for
situations like this.) Using SET XACT_ABORT ON causes the entire transaction to be aborted and rolled back if
any error is encountered. The default setting is OFF, which is consistent with ANSI-standard behavior. By setting the
option XACT_ABORT ON, we can now rerun the example that does no error checking, and no rows will be
inserted:
IF EXISTS (SELECT * FROM sysobjects WHERE name='show_error'
AND type='U')
DROP TABLE show_error
GO

CREATE TABLE show_error

(
col1 smallint NOT NULL PRIMARY KEY,
col2 smallint NOT NULL
)
GO

SET XACT_ABORT ON
BEGIN TRANSACTION

INSERT show_error VALUES (1, 1)

INSERT show_error VALUES (1, 2)
INSERT show_error VALUES (2, 2)

COMMIT TRANSACTION
GO

SELECT * FROM show_error

GO

Server: Msg 2627, Level 14, State 1, Line 1

Violation of PRIMARY KEY constraint 'PK__show_error__72C60C4A'.
Cannot insert duplicate key in object 'show_error'.

col1 col2
---- ----
(0 row(s) affected)

A final comment about constraint errors and transactions: a single data modification statement (such as an UPDATE
statement) that affects multiple rows is automatically an atomic operation, even if it's not part of an explicit transaction.
If such an UPDATE statement finds 100 rows that meet the criteria of the WHERE clause but one row fails because
of a constraint violation, no rows will be updated.

The Order of Integrity Checks

The modification of a given row will fail if any constraint is violated or if a trigger aborts the operation. As soon as a
failure in a constraint occurs, the operation is aborted, subsequent checks for that row aren't performed, and no
trigger fires for the row. Hence, the order of these checks can be important, as the following list shows.
1. Defaults are applied as appropriate.
2. NOT NULL violations are raised.
3. CHECK constraints are evaluated.
4. FOREIGN KEY checks of referencing tables are applied.
5. FOREIGN KEY checks of referenced tables are applied.
6. UNIQUE/PRIMARY KEY is checked for correctness.
7. Triggers fire.
This document is created with the unregistered version of CHM2PDF Pilot

34

Altering a Table
SQL Server 2000 allows existing tables to be modified in several ways, and you've seen some of these methods
already. You can use ALTER TABLE to add or drop constraints from a table. You've seen that some types of
constraints have the option of not being applied to existing data by using the WITH NOCHECK option. Using the
ALTER table command, you can make the following types of changes to an existing table:
 Change the datatype or NULL property of a single column.
 Add one or more new columns, with or without defining constraints for those columns.
 Add one or more constraints.
 Drop one or more constraints.
 Drop one or more columns.
 Enable or disable one or more constraints (only applies to CHECK and FOREIGN KEY constraints).
 Enable or disable one or more triggers.

Changing a Datatype
By using the ALTER COLUMN clause of ALTER TABLE, you can modify the datatype or NULL property of an
existing column. But be aware of the following restrictions:
 The modified column can't be a text, image, ntext, or rowversion (timestamp) column.
 If the modified column is the ROWGUIDCOL for the table, only DROP ROWGUIDCOL is allowed; no
datatype changes are allowed.
 The modified column can't be a computed or replicated column.
 The modified column can't have a PRIMARY KEY or FOREIGN KEY constraint defined on it.
 The modified column can't be referenced in a computed column.
 The modified column can't have the type changed to timestamp.
 If the modified column participates in an index, the only type changes that are allowed are increasing the length
of a variable-length type (for example, VARCHAR(10) to VARCHAR(20)), changing nullability of the
column, or both.
 If the modified column has a UNIQUE OR CHECK constraint defined on it, the only change allowed is
altering the length of a variable-length column. For a UNIQUE constraint, the new length must be greater than
the old length.
 If the modified column has a default defined on it, the only changes that are allowed are increasing or
decreasing the length of a variable-length type, changing nullability, or changing the precision or scale.
 The old type of the column should have an allowed implicit conversion to the new type.
 The new type always has ANSI_PADDING semantics if applicable, regardless of the current setting.
 If conversion of an old type to a new type causes an overflow (arithmetic or size), the ALTER TABLE
statement is aborted.

Here's the syntax and an example of using the ALTER COLUMN clause of the ALTER TABLE statement:

SYNTAX

ALTER TABLE table-name ALTER COLUMN column-name

{ type_name [ ( prec [, scale] ) ] [COLLATE <collation name> ]
[ NULL | NOT NULL ]
| {ADD | DROP} ROWGUIDCOL }

EXAMPLE
This document is created with the unregistered version of CHM2PDF Pilot

/* Change the length of the emp_name column in the employee

table from varchar(30) to varchar(50) */
ALTER TABLE employee
ALTER COLUMN emp_name varchar(50)

Adding a New Column

You can add a new column, with or without specifying column-level constraints. You can add only one column for
each ALTER TABLE statement. If the new column doesn't allow NULLs and isn't an identity column, the new
column must have a default constraint defined. SQL Server populates the new column in every row with a NULL, the
appropriate identity value, or the specified default. If the newly added column is nullable and has a default constraint,
the existing rows of the table are not filled with the default value, but rather with NULL values. You can override this
restriction by using the WITH VALUES clause so that the existing rows of the table are filled with the specified
default value.

Adding, Dropping, Disabling, or Enabling a

Constraint
I covered constraint modifications earlier in the discussion about constraints. The trickiest part of using ALTER
TABLE to manipulate constraints is that the word CHECK can be used in three different ways:
 To specify a CHECK constraint.
 To defer checking of a newly added constraint. In the following example, we're adding a constraint to validate
that cust_id in orders matches a cust_id in customer, but we don't want the constraint applied to existing
data:
ALTER TABLE orders
WITH NOCHECK
ADD FOREIGN KEY (cust_id) REFERENCES customer (cust_id)

NOTE
I could also use WITH CHECK to force the constraint to be applied to existing data, but that's unnecessary because
it's the default behavior.
 To enable or disable a constraint. In the next example, we're enabling all the constraints on the employee
table:
ALTER TABLE EMPLOYEE
CHECK CONSTRAINT ALL

You need to be aware when using ALTER TABLE to drop constraints that dropping a PRIMARY KEY or
UNIQUE constraint automatically drops the associated index. In fact, the only way to drop those indexes is by
altering the table to remove the constraint.
NOTE
You can't use ALTER TABLE to modify a constraint definition. You must use ALTER TABLE to drop the constraint
and then use ALTER TABLE to add a new constraint with the new definition.

Dropping a Column
You can use ALTER TABLE to remove one or more columns from a table. However, you can't drop the following
columns:
 A replicated column.
This document is created with the unregistered version of CHM2PDF Pilot

 A column used in an index.

 A column used in a CHECK, FOREIGN KEY, UNIQUE, or PRIMARY KEY constraint.
 A column associated with a default defined with the DEFAULT keyword or bound to a default object.
 A column to which a rule is bound.

Dropping a column is accomplished using the following syntax:

ALTER TABLE table-name

DROP COLUMN column-name [, next-column-name]...

NOTE
Notice the syntax difference between dropping a column and adding a new column: the word COLUMN is required
when dropping a column, but not when adding a new column to a table.

Enabling or Disabling a Trigger

You can enable or disable one or more (or all) triggers on a table using the ALTER TABLE command. I'll look at
this topic in more detail when I discuss triggers in Chapter 12.

Note that not all the ALTER TABLE variations require SQL Server to change every row when the ALTER TABLE
is issued. In many cases, SQL Server can just change the metadata (in syscolumns) to reflect the new structure. In
particular, the data isn't touched when a column is dropped, when a new column is added and NULL is assumed as
the new value for all rows, when the length of a variable-length column is changed, or when a non-nullable column is
changed to allow NULLs. All other changes to a table's structure require SQL Server to physically update every row
and to write the appropriate records to the transaction log.
This document is created with the unregistered version of CHM2PDF Pilot

34

Temporary Tables
Temporary tables are useful workspaces, like scratch pads, that you can use to try out intermediate data processing
or to share work-in-progress with other connections. You can create temporary tables from within any database, but
they exist in only the tempdb database, which is created every time the server is restarted. Don't assume that
temporary tables aren't logged: temporary tables, and actions on those tables, are logged in tempdb so that
transactions can be rolled back as necessary. However, the log isn't used for recovery of the database at system
restart because the database is entirely re-created. Likewise, tempdb is never restored from a backup, so the log in
tempdb is never needed for restoring the database.

SQL Server 2000 can log just enough information to allow rollback of transactions, without logging the additional
information that would be necessary to recover those transactions, either at system startup or when recovering from a
backup. This reduced logging means that data modification operations on tables in tempdb can be up to four times
faster than the same operations in other databases. You can use temporary tables in three ways in SQL Server:
privately, globally, and directly.

Private Temporary Tables (#)

By prefixing a table name with a single pound sign (#)—as in CREATE TABLE #my_table—you can create it (from
within any database) as a private temporary table. Only the connection that created the table can access the table,
making it truly private. Privileges can't be granted to another connection. As a temporary table, it exists for the life of
that connection only; that connection can drop the table using DROP TABLE.

Because the scoping of a private temporary table is specific to the connection that created it, you won't encounter a
name collision should you choose a table name that's used in another connection. Private temporary tables are
analogous to local variables—each connection has its own private version, and private temporary tables that are held
by other connections are irrelevant. (However, temporary tables do differ from local variables in one crucial way:
temporary tables exist for the life of the session, while local variables exist only for a single batch.)

Global Temporary Tables (##)

By prefixing a table name with double pound signs (##)—as in CREATE TABLE ##our_table—you can create a
global temporary table (from within any database and any connection). Any connection can subsequently access the
table for retrieval or data modification, even without specific permission. Unlike with private temporary tables, all
connections can use the single copy of a global temporary table. Therefore, you might encounter a name collision if
another connection has created a global temporary table of the same name, and the CREATE TABLE statement will
fail.

A global temporary table exists until the creating connection terminates and all current use of the table completes.
After the creating connection terminates, however, only those connections already accessing it are allowed to finish,
and no further use of the table is allowed. If you want a global temporary table to exist permanently, you can create
the table in a stored procedure that's marked to autostart whenever SQL Server is started. That procedure can be put
to sleep using WAITFOR and it will never terminate, so the table will never be dropped. Or you can choose to use
tempdb directly, which I'll discuss next.

Direct Use of tempdb

This document is created with the unregistered version of CHM2PDF Pilot

Realizing that tempdb is re-created every time SQL Server is started, you can use tempdb to create a table or you
can fully qualify the table name to include the database name tempdb in the CREATE TABLE statement issued from
another database. To do this, you need to establish create table privileges in tempdb.

You can set up privileges in tempdb in one of two ways every time SQL Server starts: you can set the privileges in
model (the template database) so that they are copied to tempdb when it's created at system restart, or you can have
an autostart procedure set the tempdb privileges every time SQL Server is started. One reason to consider not
setting the privileges for tempdb in the model database is because tempdb isn't the only database that will be
affected. Any new database you create will inherit those permissions as well.

Tables created directly in tempdb can exist even after the creating connection is terminated, and the creator can
specifically grant and revoke access permissions to specific users:
— Creating a table in tempdb from pubs. Another method would be
— to first do a 'use tempdb' instead of fully qualifying
— the name.
CREATE TABLE tempdb.dbo.testtemp
(
col1 int
)

Constraints on Temporary Tables

A few articles about SQL Server erroneously state that constraints don't work on temporary tables. However, all
constraints work on temporary tables explicitly built in tempdb (not using the # or ## prefixes). All constraints except
FOREIGN KEY constraints work with tables using the # (private) and ## (global) prefixes. FOREIGN KEY
references on private and global temporary tables are designed to not be enforced because such a reference could
prevent the temporary table from being dropped at close-connection time (for private temporary tables) or when a
table goes out of scope (for global temporary tables) if the referencing table wasn't dropped first.
This document is created with the unregistered version of CHM2PDF Pilot

34

System Tables
As I've mentioned, SQL Server maintains a set of tables that store information about all the objects, datatypes,
constraints, configuration options, and resources available to the SQL Server. This set of tables is sometimes called
the system catalog. We've looked at a few such tables already, in particular sysconstraints and sysobjects. Some of
the system tables exist only in the master database and contain systemwide information, and others exist in every
database (including master) and contain information about the objects and resources belonging to that particular
database.

The most common way of identifying a system table is by its name. All system tables start with the three characters
sys, but that doesn't mean that everything that starts with sys is a system table. Another way of identifying system
tables is by their object_id, which is always less than 100. A third way of identifying system tables is to look at the
type column in sysobjects; system tables have a type of S. In SQL Server 2000, some of the objects that were
system tables in previous versions have become views. You can get a list of these views by querying the sysobjects
table directly, as follows:
SELECT name FROM sysobjects
WHERE type = 'V'
AND name LIKE 'sys%'

If you run this query in the master database, you'll see six tables returned, and in all user databases you'll see only
two. The views syslogins, sysremotelogins, sysopentapes, and sysoledbusers do not exist in the user databases.

Another type of system table isn't really a table at all, but a pseudo-table. These tables take no space and are not
stored permanently on disk. Rather, SQL Server builds them dynamically every time you query them. In most cases, it
doesn't make sense to store these tables on disk because their values change constantly and exist only while the server
is running.

For example, storing syslockinfo on disk would be meaningless because locks do not exist unless SQL Server is
running. In addition, lock information needs to be accessed so often and so quickly that storing it in a table would be
much too slow. The information is actually stored in internal memory structures and is displayed as a table only when
requested. You can select from syslockinfo as if it were a table, using the following statement:
SELECT * FROM master..syslockinfo

In addition, the stored procedure sp_lock retrieves information from syslockinfo, treating it just like any other system
table. You can select from any of the tables that are really pseudo-tables, and most of them have one or more
associated stored procedures for accessing their information.

Prior to SQL Server 2000, you could find out which tables were pseudo-tables by looking in the sysindexes table,
which is where actual storage information is kept. There is a separate sysindexes table in every database, and every
table and every index has a row in sysindexes. The rows in this table indicate where the data for the table or index is
stored and how much space it takes up. Every table has a row in sysindexes with an indid (index ID) value of either
0 or 1. If the indid value is 0, the table has no clustered index and is called a heap. The value in the name column is
the name of the table. If the indid value is 1, the table has a clustered index and the name of the clustered index is
stored in the name column. The value in the id column is the ID of the table itself. Every nonclustered index on a table
has a row in sysindexes with an indid value between 2 and 250. To see how much space the table is using, all you
need to do is look at the value in the column called dpages in the row for the table. Any value greater than 0 means
the table is actually taking some space.
This document is created with the unregistered version of CHM2PDF Pilot

The following query returns space usage information for all the tables with an object ID of less than 100, which
generally means that it is a system table.
SELECT name = CONVERT(CHAR(30), o.name), rows, dpages, o.id, type
FROM sysindexes i JOIN sysobjects o ON o.id = i.id
WHERE o.id < 100 AND (indid = 0 OR indid =1)

A pseudo-system table is one that doesn't take any space. You might be tempted to look at the number of rows to
determine whether a table takes up any space, but that might not give you the right answer. In the master database on
my SQL Server, both sysreferences and sysfulltextcatalogs have 0 rows just because I haven't used those tables
yet. In earlier versions of SQL Server, you could tell that a table was a real table by looking at the value in the dpages
column. As soon as a table was created, SQL Server would allocate at least two pages for it, even without inserting
any rows. In SQL Server 2000, as you saw earlier in this chapter, that just doesn't happen. The only way to tell
whether a table is a pseudo-table is to look at the 11th bit (with a value of 1024) in the sysstat column of sysobjects.
You can perform a bit-wise AND operation of sysobjects.sysstat & 1024, which in binary would be all zeros except
for the 11th bit. Since 0 and anything is 0, all the bits except the 11th will be zero in the output. If the 11th bit is also
zero, the result will be 0, but if the 11th bit is 1, it will stay 1 in the output and the result will be 1024. The following
query will show this:
SELECT name = CONVERT(CHAR(30), o.name), rows,
PseudoTable = o.sysstat & 1024, o.type
FROM sysindexes i JOIN sysobjects o ON o.id = i.id
WHERE o.id < 100 AND (indid = 0 OR indid = 1)

The result rows that come back from the query that have a value of 1024 for the column labeled PseudoTable are
the pseudotables. Here are the pseudotables tables in my master database:
sysindexkeys
sysforeignkeys
sysmembers
sysprotects
sysperfinfo
sysprocesses
syslocks
syscurconfigs
syslockinfo
syscursorrefs
syscursors
syscursorcolumns
syscursortables
syscacheobjects
sysfiles

In the pubs database, there are only five pseudotables:

sysindexkeys
sysforeignkeys
sysmembers
sysprotects
sysfiles

There's actually a much easier way to find out if a table is a pseudotable, without doing all this fancy bit arithmetic.
SQL Server provides a whole suite of property functions, many of which I'll tell you about in relevant places
throughout this book. The function OBJECTPROPERTY has a parameter TableIsFake. The first parameter of this
function is an object ID, which you can get by using the OBJECT_ID function. The following statement returns the
This document is created with the unregistered version of CHM2PDF Pilot

TableIsFake value for the syslockinfo and sysobjects tables in the master database:
USE master
SELECT OBJECTPROPERTY(OBJECT_ID('syslockinfo'), 'TableIsFake'),
OBJECTPROPERTY(OBJECT_ID('sysobjects'), 'TableIsFake')

RESULTS:

----------- -----------
0

The return value of 1 means that the table is a pseudotable, and the return value of 0 means that it is not. The function
will return a NULL value if the table doesn't exist or if you misspell the name of the property.

Microsoft does not recommend that you directly query the system tables at all. The online documentation states:

The structure of the system catalog tables is dependent on the underlying architecture of SQL Server and changes
from one version to another. Even an application that only issues SELECT statements may have to be at least partially
rewritten when migrating to a new version of SQL Server if it directly queries system tables that change or are not
present in the new version.

So, if you aren’t supposed to directly access the system tables, how are you supposed to find out information about
your system? SQL Server 2000 provides a number of tools for accomplishing this. We took a look in the preceding
section at one of many possible property functions. There are also system stored procedures, such as sp_help and
sp_helpconstraint. I’ll tell you more about system procedures in Chapter 11. Finally, in compliance with the ANSI
SQL Standard, there is a set of INFORMATION SCHEMA VIEWS, some of which we’ll look at later in the book.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
Tables are the heart of relational databases in general and SQL Server in particular. In this chapter, we've looked at
datatype issues, including size and performance tradeoffs. SQL Server provides a datatype for almost every use. We
also looked at variable-length datatype issues and saw that it's simplistic to think that variable-length datatypes are
either always good or always bad to use. You've also seen how data is physically stored in data pages, and we
discussed some of the system table entries that are made to support tables.

SQL Server provides user-defined datatypes for support of domains, and it provides the IDENTITY property to
make a column produce autosequenced numeric values. It also provides constraints, which offer a powerful way to
ensure your data's logical integrity. In addition to the NULL/NOT NULL designation, SQL Server provides
PRIMARY KEY, UNIQUE, FOREIGN KEY, CHECK, and DEFAULT constraints. Our discussion of these
features also touched on a number of pragmatic issues, such as performance implications, conflicts between different
constraints and triggers, and transaction semantics when a constraint fails.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 7

Querying Data
You wouldn't have any reason to store data if you didn't want to retrieve it. Relational databases gained wide
acceptance principally because they enabled users to query, or access, data easily, without using predefined, rigid
navigational paths. Indeed, the acronym SQL stands for Structured Query Language. Microsoft SQL Server provides
rich and powerful query capabilities.

In this chapter, I'll examine the SELECT statement and concentrate on features beyond the basics, including NULL,
OUTER JOIN, correlated subqueries, aggregates, UNION, and the special CUBE operator. In Chapter 9, we'll see
how to modify data. Then, in Chapter 10, I'll build on these topics and explore the constructs and techniques that
make Transact-SQL more than simply a query language.
This document is created with the unregistered version of CHM2PDF Pilot

34

The SELECT Statement

The SELECT statement is the most frequently used SQL command and is the fundamental way to query data. The
SQL syntax for the SELECT statement is intuitive—at least in its simplest forms—and resembles how you might state
a request in English. As its name implies, it is also structured and precise. However, a SELECT statement can be
obscure or even tricky. Brainteasers about writing a SELECT statement to perform some tricky task often appear in
database publications. As long as you write a query that's syntactically correct, you get a result. However, it might not
be the result you really want—you might get the answer to a different question than the one you thought you posed!
Queries with incorrect semantics commonly cause bugs in applications, so you should understand proper query
formulation.

The basic form of SELECT, which uses brackets ([]) to identify optional items, is shown here.

SELECT [DISTINCT][TOP n] <columns to be chosen, optionally eliminating

duplicate rows from result set or limiting
number of rows to be returned>
[FROM] <table names>
[WHERE] <criteria that must be true for a row to be chosen>
[GROUP BY] <columns for grouping aggregate functions>
[HAVING] <criteria that must be met for aggregate functions>
[ORDER BY] <optional specification of how the results should be
sorted>

Notice that the only clause that must always be present is the verb SELECT; the other clauses are optional. For
example, if the entire table is needed, you don't need to restrict data using certain criteria, so you can omit the
WHERE clause.

Here's a simple query that retrieves all columns of all rows from the authors table in the pubs sample database:
SELECT * FROM authors

Figure 7-1 shows the output.

Using tools such as OSQL and SQL Query Analyzer, you can issue queries interactively. You can also, of course,
build queries into applications. The calling application determines the formatting of the data output. SQL Server
returns the data back to the calling application, which then works with the data. In the case of an ad hoc query tool, it
displays the output on a monitor.

Figure 7-1. The authors table in the pubs database.

NOTE
This document is created with the unregistered version of CHM2PDF Pilot

In general, this book concentrates on topics specific to Microsoft SQL Server, but it wouldn't be complete without
some discussion of the SQL language. Because the treatment of generic SQL in this book is, by design, far from
complete, please see Appendix D for suggestions on further reading, including books on ANSI Standard SQL-92
and SQL-99 as well as books on the nuances of Microsoft Transact-SQL.

The power of the SQL language begins to reveal itself when you limit the information to be returned to specified
ranges. For example, you can specify that a query from one table return only certain columns or rows that meet your
stated criteria. In the select list, you specify the exact columns you want, and then in the WHERE clause, you specify
the criteria that determine whether a row should be included in the answer set. Still using the pubs sample database,
suppose we want to find the first name and the city, state, and zip code of residence for authors whose last name is
Ringer:
SELECT au_lname, au_fname, city, state, zip
FROM authors
WHERE au_lname='Ringer'

Here's the output:

au_lname au_fname city state zip
-------- -------- -------------- ----- -----
Ringer Albert Salt Lake City UT 84152
Ringer Anne Salt Lake City UT 84152

The results of this query tell us that two authors are named Ringer. But we're interested only in Anne. Retrieving only
Anne Ringer's data requires an additional expression that is combined (AND'ed) with the original. In addition, we'd
like the output to have more intuitive names for some columns, so we respecify the query:
SELECT 'Last Name'=au_lname, 'First'=au_fname, city, state, zip
FROM authors
WHERE au_lname='Ringer' AND au_fname='Anne'

Here's the output, just as we want it:

Last Name First city state zip
--------- ----- -------------- ----- -----
Ringer Anne Salt Lake City UT 84152
This document is created with the unregistered version of CHM2PDF Pilot

34

Joins
You gain much more power when you join tables, which typically results in combining columns of matching rows to
project and return a virtual table. Usually, joins are based on the primary and foreign keys of the tables involved,
although the tables aren't required to explicitly declare keys. The pubs database contains a table of authors (authors)
and a table of book titles (titles). An obvious query would be, "Show me the titles that each author has written and
sort the results alphabetically by author. I'm interested only in authors who live outside California." Neither the
authors table nor the titles table alone has all this information. Furthermore, a many-to-many relationship exists
between authors and titles; an author might have written several books, and a book might have been written by
multiple authors. So an intermediate table, titleauthor, exists expressly to associate authors and titles, and this table is
necessary to correctly join the information from authors and titles. To join these tables, you must include all three
tables in the FROM clause of the SELECT statement, specifying that the columns that make up the keys have the
same values:
SELECT
'Author'=RTRIM(au_lname) + ', ' + au_fname,
'Title'=title
FROM authors AS A JOIN titleauthor AS TA
ON A.au_id=TA.au_id -- JOIN CONDITION
JOIN titles AS T
ON T.title_id=TA.title_id -- JOIN CONDITION
WHERE A.state <> 'CA'
ORDER BY 1

Here's the output:

Author Title
------------------------ ----------------------------------
Blotchet-Halls, Reginald Fifty Years in Buckingham Palace
Kitchens
DeFrance, Michel The Gourmet Microwave
del Castillo, Innes Silicon Valley Gastronomic Treats
Panteley, Sylvia Onions, Leeks, and Garlic: Cooking
Secrets of the Mediterranean
Ringer, Albert Is Anger the Enemy?
Ringer, Albert Life Without Fear
Ringer, Anne The Gourmet Microwave
Ringer, Anne Is Anger the Enemy?

Before discussing join operations further, let's study the preceding example. The author's last and first names have
been concatenated into one field. The RTRIM (right trim) function is used to strip off any trailing whitespace from the
au_lname column. Then we add a comma and a space and concatenate on the au_fname column. This column is
then aliased as simply Author and is returned to the calling application as a single column.
NOTE
The RTRIM function isn't needed for this example. Because the column is of type varchar, trailing blanks won't be
present. RTRIM is shown for illustration purposes only.

Another important point is that the ORDER BY 1 clause indicates that the results should be sorted by the first column
in the result set. It's more typical to use the column name rather than its number, but using the column number provides
a convenient shorthand when the column is an expression and hence isn't present in the base table or view (virtual
table) being queried.
This document is created with the unregistered version of CHM2PDF Pilot

Instead of using ORDER BY 1, you can repeat the same expression used in the select list and specify ORDER BY
RTRIM(au_lname) + ', ' + au_fname instead. Alternatively, SQL Server provides a feature supported by the ANSI
SQL-99 standard that allows sorting by columns not included in the select list. So even though you don't individually
select the columns au_lname or, au_fname, you can nonetheless choose to order the query based on these columns
by specifying columns ORDER BY au_lname, au_fname. We'll see this in the next example. Notice also that the
query contains comments (-- JOIN CONDITION). A double hyphen (--) signifies that the rest of the line is a
comment (similar to // in C++). You can also use the C-style /* comment block */, which allows blocks of comment
lines.
TIP
Comments can be nested, but you should generally try to avoid this. You can easily introduce a bug by not realizing
that a comment is nested within another comment and misreading the code.

Now let's examine the join in the example above. The ON clauses specify how the tables relate and set the join
criteria, stating that au_id in authors must equal au_id in titleauthor, and title_id in titles must equal title_id in
titleauthor. This type of join is referred to as an equijoin, and it's the most common type of join operation. To
remove ambiguity, you must qualify the columns. You can do this by specifying the columns in the form table.column,
as in authors.au_id = titleauthor.au_id. The more compact and common way to do this, however, is by specifying
a table alias in the FROM clause, as was done in this example. By following the titles table with the word AS and
the letter T, the titles table will be referred to as T anywhere else in the query where the table is referenced. Typically,
such an alias consists of one or two letters, although it can be much longer (following the same rules as identifiers).

After a table is aliased, it must be referred to by the alias, so now we can't refer to authors.au_id because the
authors table has been aliased as A. We must use A.au_id. Note also that the state column of authors is referred to
as A.state. The other two tables don't contain a state column, so qualifying it with the A. prefix isn't necessary;
however, doing so makes the query more readable—and less prone to subsequent bugs.

The join is accomplished using the ANSI JOIN SQL syntax, which was introduced in SQL Server version 6.5.
Many examples and applications continue to use an old-style JOIN syntax, which is shown below. (The term
"old-style JOIN" is actually used by the SQL-92 specification.) The ANSI JOIN syntax is based on ANSI SQL-92.
The main differences between the two types of join formulations are
 The ANSI JOIN actually uses the keyword JOIN.
 The ANSI JOIN segregates join conditions from search conditions.

The ANSI JOIN syntax specifies the JOIN conditions in the ON clauses (one for each pair of tables), and the search
conditions are specified in the WHERE clause—for example, WHERE state <> 'CA'. Although slightly more
verbose, the explicit JOIN syntax is more readable. There's no difference in performance; behind the scenes, the
operations are the same. Here's how you can respecify the query using the old-style JOIN syntax:
SELECT
'Author'=RTRIM(au_lname) + ', ' + au_fname,
'Title'=title
FROM authors A, titles T, titleauthor TA
WHERE
A.au_id=TA.au_id AND T.title_id=TA.title_id -- JOIN CONDITIONS
AND A.state <> 'CA'
ORDER BY 1

This query produces the same output and the same execution plan as the previous query.

One of the most common errors that new SQL users make when using the old-style JOIN syntax is not specifying
the join condition. Omitting the WHERE clause is still a valid SQL request and causes a result set to be returned.
However, that result set is likely not what the user wanted. In the query above, omitting the WHERE clause would
return the Cartesian product of the three tables: it would generate every possible combination of rows between them.
Although in a few unusual cases you might want all permutations, usually this is just a user error. The number of rows
This document is created with the unregistered version of CHM2PDF Pilot

returned can be huge and typically doesn't represent anything meaningful. For example, the Cartesian product of the
three small tables here (authors, titles, and titleauthor each have less than 26 rows) generates 10,350 rows of
(probably) meaningless output.

Using the ANSI JOIN syntax, it's impossible to accidentally return a Cartesian product of the tables—and that's one
reason to use ANSI JOINs almost exclusively. The ANSI JOIN syntax requires an ON clause for specifying how the
tables are related. In cases where you actually do want a Cartesian product, the ANSI JOIN syntax allows you to
use a CROSS JOIN operator, which we'll examine in more detail later in this chapter.
NOTE
This book uses the ANSI JOIN syntax almost exclusively. You might want to try translating some of them into
queries using old-style JOIN syntax, because you should be able to recognize both forms. If you have to read SQL
code written earlier than version 7, you're bound to come across queries using this older syntax.

The most common form of join is an equijoin, which means that the condition linking the two tables is based on
equality. An equijoin is sometimes referred to as aninner join to differentiate it from an outer join, which I'll discuss
shortly. Strictly speaking, an inner join isn't quite the same as an equijoin; an inner join can use an operator such as
less than (<) or greater than (>). So all equijoins are inner joins but not all inner joins are equijoins. I'll use the term
inner join because it is the broader term. To make this distinction clear in your code, you can use the INNER JOIN
syntax in place of JOIN, as in this example:
FROM authors AS A INNER JOIN titleauthor TA ON A.au_id=TA.au_id

Other than making the syntax more explicit, there's no difference in the semantics or the execution. By convention, the
modifier INNER generally isn't used.
NOTE
ANSI SQL-92 also specifies the natural join operation, in which you don't have to specify the tables' column names.
By specifying syntax such as FROM authors NATURAL JOIN titleauthor, the system automatically knows how to
join the tables without your specifying the column names to match. SQL Server doesn't yet support this feature.

The ANSI specification calls for the natural join to be resolved based on identical column names between the tables.
Perhaps a better way to do this would be based on a declared primary key-foreign key relationship, if it exists.
Admittedly, declared key relationships have issues, too, because there's no restriction that only one such foreign key
relationship be set up. Also, if the natural join were limited to only such relationships, all joins would have to be known
in advance—as in the old CODASYL days.

The FROM clause in this example shows an alternative way to specify a table alias—by omitting the AS. The use of
AS preceding the table alias, as used in the previous example, conforms to ANSI SQL-92. From SQL Server's
standpoint, the two methods are equivalent (stating FROM authors AS A is identical to stating FROM authors A).
Commonly, the AS formulation is used in ANSI SQL-92 join operations and the formulation that omits AS is used
with the old-style join formulation. However, you can use either formulation—it's strictly a matter of preference.

Outer Joins
Inner joins return only rows from the respective tables that meet the conditions specified in the ON clause. In other
words, if a row in the first table doesn't match any rows in the second table, that row isn't returned in the result set. In
contrast, outer joins preserve some or all of the unmatched rows. To illustrate how easily subtle semantic errors can
be introduced, let's refer back to the previous two query examples, in which we want to see the titles written by all
authors not living in California. The result omits two writers who do not, in fact, live in California. Are the queries
wrong? No! They perform exactly as written—we didn't specify that authors who currently have no titles in the
database should be included. The results of the query are as we requested. In fact, the authors table has four rows
that have no related row in the titleauthor table, which means these "authors" actually haven't written any books.
Two of these authors don't contain the value CA in the state column, as the following result set shows.
This document is created with the unregistered version of CHM2PDF Pilot

au_id Author state

----------- ------------------- -----
341-22-1782 Smith, Meander KS
527-72-3246 Greene, Morningstar TN
724-08-9931 Stringer, Dirk CA
893-72-1158 McBadden, Heather CA

The titles table has one row for which there is no author in our database, as shown here. (Later in this chapter, you'll
see the types of queries used to produce these two result sets.)
title_id title
-------- ----------------------------------
MC3026 The Psychology of Computer Cooking

If our queries were meant to be "Show me the titles that each author has written; include authors for whom there's no
title currently in our database; if there's a title with no author, show me that as well; sort the results alphabetically by
the author; I'm interested only in authors who live outside of California," the query would use an outer join so that
authors with no matching titles would be selected:
SELECT
'Author'=RTRIM(au_lname) + ', ' + au_fname,
'Title'=title
FROM
( -- JOIN CONDITIONS
-- FIRST join authors and titleauthor
(authors AS A
FULL OUTER JOIN titleauthor AS TA ON A.au_id=TA.au_id
)
-- The result of the previous join is then joined to titles
FULL OUTER JOIN titles AS T ON TA.title_id=T.title_id
)
WHERE
state <> 'CA' OR state IS NULL
ORDER BY 1

Here's the output:

Author Title
------------------------ ----------------------------------
NULL The Psychology of Computer Cooking
Blotchet-Halls, Reginald Fifty Years in Buckingham Palace
Kitchens
DeFrance, Michel The Gourmet Microwave
del Castillo, Innes Silicon Valley Gastronomic Treats
Greene, Morningstar NULL
Panteley, Sylvia Onions, Leeks, and Garlic: Cooking
Secrets of the Mediterranean
Ringer, Albert Is Anger the Enemy?
Ringer, Albert Life Without Fear
Ringer, Anne The Gourmet Microwave
Ringer, Anne Is Anger the Enemy?
Smith, Meander NULL

The query demonstrates a full outer join. Rows in the authors and titles tables that don't have a corresponding
entry in titleauthor are still presented, but with a NULL entry for the title or author column. (The data from the
authors table is requested as a comma between the last names and the first names. Because we have the option
CONCAT_NULL_YIELDS_NULL set to ON, the result is NULL for the author column. If we didn't have that
This document is created with the unregistered version of CHM2PDF Pilot

option set to ON, SQL Server would have returned the single comma between the nonexistent last and first names.)
A full outer join preserves nonmatching rows from both the lefthand and righthand tables. In the example above, the
authors table is presented first, so it is the lefthand table when joining to titleauthor. The result of that join is the
lefthand table when joining to titles.
ANSI Outer Joins
At the beginning of this chapter, I promised not to go into much detail about generic SQL operations. The outer join
formulations here are standard ANSI SQL-92. However, this is one area in which there aren't enough good
examples. Using OUTER JOIN with more than two tables introduces some obscure issues; in fact, few products
provide support for full outer joins. Because you must understand outer joins thoroughly if you are to use OUTER
JOIN, I'll go into a bit more detail in this section.

You can generate missing rows from one or more of the tables by using either a left outer join or a right outer join.
So if we want to preserve all authors and generate a row for all authors who have a missing title, but we don't want to
preserve titles that have no author, we can reformulate the query using LEFT OUTER JOIN, as shown below. This
join preserves entries only on the lefthand side of the join. Note that the left outer join of the authors and titleauthor
columns generates two such rows (for Greene and Smith). The result of the join is the lefthand side of the join to titles
; therefore, the LEFT OUTER JOIN must be specified again to preserve these rows with no matching titles rows.

FROM
(
(authors as A
LEFT OUTER JOIN titleauthor AS TA ON A.au_id=TA.au_id
)
LEFT OUTER JOIN titles AS T ON TA.title_id=T.title_id
)

Here's the output:

Author Title
------------------------ ----------------------------------
Blotchet-Halls, Reginald Fifty Years in Buckingham Palace
Kitchens
DeFrance, Michel The Gourmet Microwave
del Castillo, Innes Silicon Valley Gastronomic Treats
Greene, Morningstar NULL
Panteley, Sylvia Onions, Leeks, and Garlic: Cooking
Secrets of the Mediterranean
Ringer, Albert Is Anger the Enemy?
Ringer, Albert Life Without Fear
Ringer, Anne The Gourmet Microwave
Ringer, Anne Is Anger the Enemy?
Smith, Meander NULL

The query produces the same rows as the full outer join, except for the row for The Psychology of Computer
Cooking. Because we specified only LEFT OUTER JOIN, there was no request to preserve titles (righthand) rows
with no matching rows in the result of the join of authors and titleauthor.

You must use care with OUTER JOIN operations because the order in which tables are joined affects which rows
are preserved and which aren't. In an inner join, the symmetric property holds (if A equals B, then B equals A) and no
difference results, whether something is on the left or the right side of the equation, and no difference results in the
order in which joins are specified. This is definitely not the case for OUTER JOIN operations. For example, consider
the following two queries and their results:

QUERY 1
This document is created with the unregistered version of CHM2PDF Pilot

SELECT
'Author'= RTRIM(au_lname) + ', ' + au_fname,
'Title'=title
FROM (titleauthor AS TA
RIGHT OUTER JOIN authors AS A ON (A.au_id=TA.au_id))
FULL OUTER JOIN titles AS T ON (TA.title_id=T.title_id)
WHERE
A.state <> 'CA' or A.state is NULL
ORDER BY 1
Author Title
------------------------ ----------------------------------
NULL The Psychology of Computer Cooking
Blotchet-Halls, Reginald Fifty Years in Buckingham Palace
Kitchens
DeFrance, Michel The Gourmet Microwave
del Castillo, Innes Silicon Valley Gastronomic Treats
Greene, Morningstar NULL
Panteley, Sylvia Onions, Leeks, and Garlic: Cooking
Secrets of the Mediterranean
Ringer, Albert Is Anger the Enemy?
Ringer, Albert Life Without Fear
Ringer, Anne The Gourmet Microwave
Ringer, Anne Is Anger the Enemy?
Smith, Meander NULL

This query produces results semantically equivalent to the previous FULL OUTER JOIN formulation, although we've
switched the order of the authors and titleauthor tables. This query and the previous one preserve both authors with
no matching titles and titles with no matching authors. This might not be obvious because RIGHT OUTER JOIN is
clearly different than FULL OUTER JOIN. However, in this case we know it's true because a FOREIGN KEY
constraint exists on the titleauthor table to ensure that there can never be a row in the titleauthor table that doesn't
match a row in the authors table, and the FOREIGN KEY columns in titleauthor are defined to not allow NULL
values. So we can be confident that the titleauthor RIGHT OUTER JOIN to authors can't produce any fewer rows
than would a FULL OUTER JOIN.

But if we modify the query ever so slightly by changing the join order again, look what happens:

QUERY 2
SELECT
'Author'=rtrim(au_lname) + ', ' + au_fname,
'Title'=title
FROM (titleauthor AS TA
FULL OUTER JOIN titles AS T ON TA.title_id=T.title_id)
RIGHT OUTER JOIN authors AS A ON A.au_id=TA.au_id
WHERE
A.state <> 'CA' or A.state is NULL
ORDER BY 1
Author Title
------------------------ ----------------------------------
Blotchet-Halls, Reginald Fifty Years in Buckingham Palace
Kitchens
DeFrance, Michel The Gourmet Microwave
del Castillo, Innes Silicon Valley Gastronomic Treats
Greene, Morningstar NULL
Panteley, Sylvia Onions, Leeks, and Garlic: Cooking
Secrets of the Mediterranean
Ringer, Albert Is Anger the Enemy?
Ringer, Albert Life Without Fear
Ringer, Anne The Gourmet Microwave
Ringer, Anne Is Anger the Enemy?
Smith, Meander NULL
This document is created with the unregistered version of CHM2PDF Pilot

At a glance, Query 2 looks equivalent to Query 1, although the join order is slightly different. But notice how the
results differ. Query 2 didn't achieve the goal of preserving the titles rows without corresponding authors, and the
row for The Psychology of Computer Cooking is again excluded. This row would have been preserved in the first
join operation:
FULL OUTER JOIN titles AS T ON TA.title_id=T.title_id

But then the row is discarded because the second join operation preserves only authors without matching titles:
RIGHT OUTER JOIN authors AS A ON A.au_id=TA.au_id

Because the title row for The Psychology of Computer Cooking is on the lefthand side of this join operation and
only a RIGHT OUTER JOIN operation is specified, this title is discarded.
NOTE
Just as INNER is an optional modifier, so is OUTER. Hence, LEFT OUTER JOIN can be equivalently specified as
LEFT JOIN, and FULL OUTER JOIN can be equivalently expressed as FULL JOIN. However, although INNER
is seldom used by convention and is usually only implied, OUTER is almost always used by convention when
specifying any type of outer join.

Because join order matters, I urge you to use parentheses and indentation carefully when specifying OUTER JOIN
operations. Indentation, of course, is always optional, and use of parentheses is often optional. But as this example
shows, it's easy to make mistakes that result in your queries returning the wrong answers. As is true with almost all
programming, simply getting into the habit of using comments, parentheses, and indentation often results in such bugs
being noticed and fixed by a developer or database administrator before they make their way into your applications.

The Obsolete *= OUTER JOIN Operator

Prior to version 6.5, SQL Server had limited outer-join support in the form of the special operators *= and =*.
Many people assume that the LEFT OUTER JOIN syntax is simply a synonym for *=, but this isn't the case. LEFT
OUTER JOIN is semantically different from and superior to *=.

For inner joins, the symmetric property holds, so the issues with old-style JOIN syntax don't exist. You can use
either new-style or old-style syntax with inner joins. For outer joins, you should consider the *= operator obsolete
and move to the OUTER JOIN syntax as quickly as possible—the *= operator might be dropped entirely in future
releases of SQL Server.

ANSI's OUTER JOIN syntax, which was adopted in SQL Server version 6.5, recognizes that for outer joins, the
conditions of the join must be evaluated separately from the criteria applied to the rows that are joined. ANSI gets it
right by separating the JOIN criteria from the WHERE criteria. The old SQL Server *= and =* operators are prone
to ambiguities, especially when three or more tables, views, or subqueries are involved. Often the results aren't what
you'd expect, even though you might be able to explain them. But sometimes you simply can't get the result you want.
These aren't implementation bugs; more accurately, these are inherent limitations in trying to apply the outer-join
criteria in the WHERE clause.
Who's Writing the Queries?
Notice that I use the terms "developer" and "database administrator," not "end user." SQL is often regarded as a
language suitable for end users as well as database professionals. This is true to some extent, such as with
straightforward single table queries. But if you expect end users to understand the semantics of outer joins or deal
correctly with NULL, you might as well just give them a random data generator. These concepts are tricky, and it's
your job to insulate your users from them. Later in this chapter, you'll see how you can use views to accomplish this.

When *= was introduced, no ANSI specification existed for OUTER JOIN or even for INNER JOIN. Just the
This document is created with the unregistered version of CHM2PDF Pilot

old-style join existed, with operators such as =* in the WHERE clause. So the designers quite reasonably tried to fit
an outer-join operator into the WHERE clause, which was the only place where joins were ever stated. However,
efforts to resolve this situation helped spur the ANSI SQL committee's specification of proper OUTER JOIN syntax
and semantics. Implementing outer joins correctly is difficult, and SQL Server is one of the few mainstream products
that has done so.

To illustrate the semantic differences and problems with the old *= syntax, I'll walk through a series of examples using
both new and old outer-join syntax. The following is essentially the same outer-join query shown earlier, but this one
returns only a count. It correctly finds the 11 rows, preserving both authors with no titles and titles with no authors.
-- New style OUTER JOIN correctly finds 11 rows
SELECT COUNT(*)
FROM
(
-- FIRST join authors and titleauthor
(authors AS A
LEFT OUTER JOIN titleauthor AS TA ON A.au_id=TA.au_id
)

-- The result of the previous join is then joined to titles

FULL OUTER JOIN titles AS T ON TA.title_id=T.title_id
)
WHERE
state <> 'CA' OR state IS NULL

There's really no way to write this query—which does a full outer join—using the old syntax, because the old syntax
simply isn't expressive enough. Here's what looks to be a reasonable try—but it generates several rows that you
wouldn't expect:
-- Attempt with old-style join. Really no way to do FULL OUTER
-- JOIN. This query finds 144 rows--WRONG!!
SELECT COUNT(*)
FROM
authors A, titleauthor TA, titles T
WHERE
A.au_id *= TA.au_id
AND
TA.title_id =* T.title_id
AND
(state <> 'CA' OR state IS NULL)

Now let's examine some issues with the old-style outer join using a simple example of only two tables, Customers
and Orders:
CREATE TABLE Customers
(
Cust_ID int PRIMARY KEY,
Cust_Name char(20)
)

CREATE TABLE Orders

(
OrderID int PRIMARY KEY,
Cust_ID int REFERENCES Customers(Cust_ID)
)
GO

INSERT Customers VALUES (1, 'Cust 1')

INSERT Customers VALUES (2, 'Cust 2')
INSERT Customers VALUES (3, 'Cust 3')
This document is created with the unregistered version of CHM2PDF Pilot

INSERT Orders VALUES (10001, 1)

INSERT Orders VALUES (20001, 2)
GO

At a glance, in the simplest case, the new-style and old-style syntax appear to work the same. Here's the new syntax:
SELECT
'Customers.Cust_ID'=Customers.Cust_ID, Customers.Cust_Name,
'Orders.Cust_ID'=Orders.Cust_ID
FROM Customers LEFT JOIN Orders
ON Customers.Cust_ID=Orders.Cust_ID

Here's the output:

Customers.Cust_ID Cust_Name Orders.Cust_ID
----------------- --------- --------------
1 Cust 1 1
2 Cust 2 2
3 Cust 3 NULL

And here's the old-style syntax:

SELECT 'Customers.Cust_ID'=Customers.Cust_ID, Customers.Cust_Name,
'Orders.Cust_ID'=Orders.Cust_ID
FROM Customers, Orders WHERE Customers.Cust_ID *= Orders.Cust_ID

Here's the output:

Customers.Cust_ID Cust_Name Orders.Cust_ID
----------------- --------- --------------
1 Cust 1 1
2 Cust 2 2
3 Cust 3 NULL

But as soon as you begin to add restrictions, things get tricky. What if you want to filter out Cust 2? With the new
syntax it's easy, but remember not to filter out the row with NULL that the outer join just preserved!
SELECT 'Customers.Cust_ID'=Customers.Cust_ID, Customers.Cust_Name,
'Orders.Cust_ID'=Orders.Cust_ID
FROM Customers LEFT JOIN Orders
ON Customers.Cust_ID=Orders.Cust_ID
WHERE Orders.Cust_ID <> 2 OR Orders.Cust_ID IS NULL

Here's the output:

Customers.Cust_ID Cust_Name Orders.Cust_ID
----------------- --------- --------------
1 Cust 1 1
3 Cust 3 NULL

Now try to do this query using the old-style syntax and filter out Cust 2:
SELECT 'Customers.Cust_ID'=Customers.Cust_ID, Customers.Cust_Name,
'Orders.Cust_ID'=Orders.Cust_ID
FROM Customers, Orders
This document is created with the unregistered version of CHM2PDF Pilot

WHERE Customers.Cust_ID *= Orders.Cust_ID

AND (Orders.Cust_ID <> 2 OR Orders.Cust_ID IS NULL)

Here's the output:

Customers.Cust_ID Cust_Name Orders.Cust_ID
----------------- --------- --------------
1 Cust 1 1
2 Cust 2 NULL
3 Cust 3 NULL

Notice that this time, we don't get rid of Cust 2. The check for NULL occurs before the JOIN, so the outer-join
operation puts Cust 2 back. This result might be less than intuitive, but at least we can explain and defend it. That's
not always the case, as you'll see in a moment.

If you look at the preceding query, you might think that we should have filtered out Customers.Cust_ID rather than
Orders.Cust_ID. How did we miss that? Surely this query will fix the problem:
SELECT 'Customers.Cust_ID'=Customers.Cust_ID, Customers.Cust_Name,
'Orders.Cust_ID'=Orders.Cust_ID
FROM Customers, Orders
WHERE Customers.Cust_ID *= Orders.Cust_ID
AND (Customers.Cust_ID <> 2 OR Orders.Cust_ID IS NULL)

Here's the output:

Customers.Cust_ID Cust_Name Orders.Cust_ID
----------------- --------- --------------
1 Cust 1 1
2 Cust 2 NULL
3 Cust 3 NULL

Oops! Same result. The problem here is that Orders.Cust_ID IS NULL is now being applied after the outer join, so
the row is presented again. If we're careful and understand exactly how the old outer join is processed, we can get the
results we want with the old-style syntax for this query. We need to understand that the OR Orders.Cust_ID IS
NULL puts back Cust_ID 2, so just take that out. Here is the code:
SELECT 'Customers.Cust_ID'=Customers.Cust_ID, Customers.Cust_Name,
'Orders.Cust_ID'=Orders.Cust_ID
FROM Customers, Orders
WHERE Customers.Cust_ID *= Orders.Cust_ID
AND Customers.Cust_ID <> 2

Here's the output:

Customers.Cust_ID Cust_Name Orders.Cust_ID
----------------- --------- --------------
1 Cust 1 1
3 Cust 3 NULL

Finally! This is the result we want. And if you really think about it, the semantics are even understandable (although
different from the new style). So maybe this is much ado about nothing; all we have to do is understand how it works,
right? Wrong. Besides the issues of joins with more than two tables and the lack of a full outer join, we also can't
effectively deal with subqueries and views (virtual tables). For example, let's try creating a view with the old-style
This document is created with the unregistered version of CHM2PDF Pilot

outer join:
CREATE VIEW Cust_old_OJ AS
(SELECT Orders.Cust_ID, Customers.Cust_Name
FROM Customers, Orders
WHERE Customers.Cust_ID *= Orders.Cust_ID)

A simple select from the view looks fine:

SELECT * FROM Cust_old_OJ

And it gives this output:

Cust_ID Cust_Name
------- ---------
1 Cust 1
2 Cust 2
NULL Cust 3

But restricting from this view doesn't seem to make sense:

SELECT * FROM Cust_old_OJ WHERE Cust_ID <> 2
AND Cust_ID IS NOT NULL

The output shows NULLs in the Cust_ID column, even though we tried to filter them out:
Cust_ID Cust_Name
------- ---------
1 Cust 1
NULL Cust 2
NULL Cust 3

If we expand the view to the full select and we realize that Cust_ID is Orders.Cust_ID, not Customers.Cust_ID,
perhaps we can understand why this happened. But we still can't filter out those rows!

In contrast, if we create the view with the new syntax and correct semantics, it works exactly as expected:
CREATE VIEW Cust_new_OJ AS
(SELECT Orders.Cust_ID, Customers.Cust_Name
FROM Customers LEFT JOIN Orders
ON Customers.Cust_ID=Orders.Cust_ID )
GO

SELECT * FROM Cust_new_OJ WHERE Cust_ID <> 2 AND Cust_ID IS NOT NULL

Here's what was expected:

Cust_ID Cust_Name
------- ---------
1 Cust 1

In the preceding examples, the new syntax performed the outer join and then applied the restrictions in the WHERE
clause to the result. In contrast, the old style applied the WHERE clause to the tables being joined and then
performed the outer join, which can reintroduce NULL rows. This is why the results often seemed bizarre. However,
This document is created with the unregistered version of CHM2PDF Pilot

if that behavior is what you want, you could apply the criteria in the JOIN clause instead of in the WHERE clause.

The following example uses the new syntax to mimic the old behavior. The WHERE clause is shown here simply as a
placeholder to make clear that the statement Cust_ID <> 2 is in the JOIN section, not in the WHERE section.
SELECT 'Customers.Cust_ID'=Customers.Cust_ID, Customers.Cust_Name,
'Orders.Cust_ID'=Orders.Cust_ID
FROM Customers LEFT JOIN Orders
ON Customers.Cust_ID=Orders.Cust_ID
AND Orders.Cust_ID <> 2
WHERE 1=1

Here's the output:

Customers.Cust_ID Cust_Name Orders.Cust_ID
----------------- --------- --------------
1 Cust 1 1
2 Cust 2 NULL
3 Cust 3 NULL

As you can see, the row for Cust 2 was filtered out from the Orders table before the join, but because it was NULL,
it was reintroduced by the OUTER JOIN operation.

With the improvements in outer-join support, you can now use outer joins where you couldn't previously. A bit later,
you'll see how to use an outer join instead of a correlated subquery in a common type of query.

Cross Joins
In addition to INNER JOIN, OUTER JOIN, and FULL JOIN, the ANSI JOIN syntax allows a CROSS JOIN.
Earlier, you saw that the advantage of using the ANSI JOIN syntax was that you wouldn't accidentally create a
Cartesian product. However, in some cases, creating a Cartesian product might be exactly what you want to do.
SQL Server allows you to specify a CROSS JOIN with no ON clause to produce a Cartesian product.

For example, one use for CROSS JOINs is to generate sample or test data. For example, to generate 10,000 names
for a sample employees table, you don't have to come up with 10,000 individual INSERT statements. All you need to
do is build a first_names table and a last_names table with 26 names each (perhaps one starting with each letter of
the English alphabet), and a middle_initials table with the 26 letters. When these three small tables are joined using
the CROSS JOIN operator, the result is well over 10,000 unique names to insert into the employees table. The
SELECT statement used to generate these names looks like this:
SELECT first_name, middle_initial, last_name
FROM first_names CROSS JOIN middle_initials
CROSS JOIN last_names

To summarize the five types of ANSI JOIN operations, consider two tables, TableA and TableB:

INNER JOIN (default)

TableA INNER JOIN TableB ON join_condition

The INNER JOIN returns rows from either table only if they have a corresponding row in the other table. In other
words, the INNER JOIN disregards any rows in which the specific join condition, as specified in the ON clause, isn't
met.
This document is created with the unregistered version of CHM2PDF Pilot

LEFT OUTER JOIN

TableA LEFT OUTER JOIN TableB ON join_condition

The LEFT OUTER JOIN returns all rows for which a connection exists between TableA and TableB; in addition, it
returns all rows from TableA for which no corresponding row exists in TableB. In other words, it preserves
unmatched rows from TableA. TableA is sometimes called the preserved table. In result rows containing unmatched
rows from TableA, any columns selected from TableB are returned as NULL.

RIGHT OUTER JOIN

TableA RIGHT OUTER JOIN TableB ON join_condition

The RIGHT OUTER JOIN returns all rows for which a connection exists between TableA and TableB; in addition,
it returns all rows from TableB for which no corresponding row exists in TableA. In other words, it preserves
unmatched rows from TableB, and in this case TableB is the preserved table. In result rows containing unmatched
rows from TableB, any columns selected from TableA are returned as NULL.

FULL OUTER JOIN

TableA FULL OUTER JOIN TableB ON join_condition

The FULL OUTER JOIN returns all rows for which a connection exists between TableA and TableB. In addition, it
returns all rows from TableA for which no corresponding row exists in TableB, with any values selected from TableB
returned as NULL. In addition, it returns all rows from TableB for which no corresponding row exists in TableA, with
any values selected from TableA returned as NULL. In other words, FULL OUTER JOIN acts as a combination of
LEFT OUTER JOIN and RIGHT OUTER JOIN.

CROSS JOIN
TableA CROSS JOIN TableB

The CROSS JOIN returns all rows from TableA combined with all rows from TableB. No ON clause exists to
indicate any connecting column between the tables. A CROSS JOIN returns a Cartesian product of the two tables.
This document is created with the unregistered version of CHM2PDF Pilot

34

Dealing with NULL

The full outer join example shown in "The Obsolete *= OUTER JOIN Operator" has a necessary twist because of
the NULL value that's generated for the missing author state column. Note the WHERE clause:
WHERE state <> 'CA' OR state IS NULL

Without the following statement

OR state IS NULL

our favorite row, which contains the title The Psychology of Computer Cooking, wouldn't have been selected.
Because this title is missing a corresponding author, the state column for the query is NULL.

The expression
state <> 'CA'

doesn't evaluate to TRUE for a NULL value. NULL is considered unknown. If you were to omit the expression
OR state IS NULL

and execute the query once with the predicate

state = 'CA'

and once with the predicate

state <> 'CA'

it might seem intuitive that, between the two queries, all rows will have been selected. But this isn't the case. A
NULL state satisfies neither condition, and the row with the title The Psychology of Computer Cooking won't be
part of either result.

This example illustrates the essence of three-valued logic, which is so named because a comparison can evaluate to
one of three conditions: TRUE, FALSE, or Unknown. Dealing with NULL entries requires three-valued logic. If you
allow NULL entries or have cases in which outer joins are necessary, correctly applying three-valued logic is
necessary to avoid introducing bugs into your application.

Assume that the table Truth_Table has columns X and Y. We have only one row in the table, as follows, with X
being NULL and Y being 1:
X Y
---- -
NULL 1
This document is created with the unregistered version of CHM2PDF Pilot

Without looking ahead, test yourself to see which of the following queries returns 1:
SELECT 1 FROM Truth_Table WHERE X <> Y -- NOT EQUAL
SELECT 1 FROM Truth_Table WHERE X = Y -- EQUAL
SELECT 1 FROM Truth_Table WHERE X != Y -- NOT EQUAL
-- (alternative formulation)
SELECT 1 FROM Truth_Table WHERE X < Y -- LESS THAN
SELECT 1 FROM Truth_Table WHERE X !< Y -- NOT LESS THAN
SELECT 1 FROM Truth_Table WHERE X > Y -- GREATER THAN
SELECT 1 FROM Truth_Table WHERE X !> Y -- NOT GREATER THAN

In fact, none of them returns 1 because none of the comparison operations against X, which is NULL, evaluate to
TRUE. All evaluate to Unknown. Did you get this right? If so, are you confident that all your developers would, too?
If you allow NULL, your answer had better be yes.

Further complexity arises when you use expressions that can be Unknown with the logical operations of AND, OR,
and NOT. Given that an expression can be TRUE, FALSE, or Unknown, the truth tables in Table 7-1 summarize the
result of logically combining expressions.

These truth tables can reveal some situations that might not be obvious otherwise. For example, you might assume
that the condition (X >= 0 OR X <= 0) must be TRUE because X for any number always evaluates to TRUE; it must
be either 0 or greater, or 0 or less. However, if X is NULL, then X >= 0 is Unknown, and X <= 0 is also Unknown.
Therefore, the expression evaluates as Unknown OR Unknown. As the OR truth table shows, this evaluates to
Unknown. To further illustrate, the following SELECT statement doesn't return a 1:
SELECT 1 FROM Truth_Table WHERE (X >= 0 OR X <= 0)

Table 7-1. Truth tables showing results of logically combining expressions.

AND with Value True Unknown False
TRUE TRUE Unknown FALSE
Unknown Unknown Unknown FALSE
FALSE FALSE FALSE FALSE
OR with Value True Unknown False
TRUE TRUE TRUE TRUE
Unknown TRUE Unknown Unknown
FALSE TRUE Unknown FALSE
NOT Evaluates to
TRUE FALSE
Unknown Unknown
False TRUE

And because (X >= 0 OR X <= 0) is Unknown, the condition:

WHERE NOT (X >= 0 OR X <= 0)

is equivalent (in pseudocode) to:

WHERE NOT (Unknown)

The NOT truth table shows that NOT (Unknown) is Unknown, so the following negation of the previous SELECT
statement also returns nothing:
This document is created with the unregistered version of CHM2PDF Pilot

SELECT 1 FROM Truth_Table WHERE NOT (X >= 0 OR X <= 0)

And, at the risk of belaboring the point, since neither expression evaluates to TRUE, OR'ing them makes no
difference either; consequently, this SELECT also returns nothing:
SELECT 1 FROM TRUTH_TABLE
WHERE (X >= 0 OR X <= 0) OR NOT (X >= 0 OR X <= 0)

The fact that none of these SELECT statements evaluates to TRUE, even when negating the expression, illustrates
that not evaluating to TRUE doesn't imply evaluating to FALSE. Rather, it means either evaluating to FALSE or
evaluating to Unknown.

Two special operators, IS NULL and IS NOT NULL, exist to deal with NULL. Checking an expression with one
of these operators always evaluates to either TRUE or FALSE. The result returned by IS NULL depends on whether
the original expression is a known or unknown value. No Unknown condition is produced when you use IS NULL or
IS NOT NULL, as shown in Table 7-2.

Table 7-2. Truth tables for IS NULL and IS NOT NULL.

IS NULL Evaluates to IS NOT NULL Evaluates to
Known Value FALSE Known Value TRUE
Unknown Value TRUE Unknown Value FALSE
(NULL) (NULL)

In the full outer-join query example, the search criteria were

WHERE state <> 'CA' OR state IS NULL

For the row with the title The Psychology of Computer Cooking (which was produced by the outer join operation
preserving the titles row with an unmatched authors entry), the authors.state column is NULL. Hence, the
expression
state <> 'CA'

is Unknown for that row, but the expression

state IS NULL

is TRUE for that row. The full expression evaluates to (Unknown OR TRUE), which is TRUE, and the row qualifies.
Without
OR state IS NULL

the expression is only (Unknown), so the row doesn't qualify.

NULL in the Real World

As you know, I strongly recommend that you minimize the use of NULL. In outer-join operations, you should
carefully account for NULL values that are generated to preserve rows that don't have a match in the table being
joined. And even if you're comfortable with three-valued logic, your developers and anyone querying the data using
This document is created with the unregistered version of CHM2PDF Pilot

the SQL language typically won't be; therefore, introducing NULL will be a source of bugs just waiting to happen.
You can, of course, make all column definitions NOT NULL and declare default values if no value is specified.

You can often avoid the problems of having no matching rows—without the need for outer-join operations—by
providing placeholder, or dummy, rows. For example, you can easily create a dummy titles row and a dummy
authors row. By using constraints or triggers, you can insist that every row in authors must have a matching entry in
titleauthor, even if the ultimate matching row in titles is simply the dummy row. Likewise, every row in titles can be
made to reference titleauthor, even if the matching row in authors is just the dummy row. In this way, a standard
inner join returns all rows because no unmatched rows exist. There's no outer join or NULL complication.

Here's an example: after you add a dummy row for authors, one for titles, and one for each unmatched row to the
titleauthor table, you use a standard inner join (with INNER specified purely to emphasize the point) to produce
essentially the same results as the full outer join. The only difference in the results is that the placeholder values ***No
Current Author*** and ***No Current Title*** are returned instead of NULL.
BEGIN TRAN -- Transaction will be rolled back so as to avoid
-- permanent changes

-- Dummy authors row

INSERT authors
(au_id, au_lname, au_fname, phone, address, city, state, zip, contract)
VALUES
('000-00-0000', '***No Current Author***',
'', 'NONE', 'NONE', 'NONE', 'XX', '99999', 0)

-- Dummy titles row

INSERT titles
(title_id, title, type, pub_id, price, advance, royalty, ytd_sales,
notes, pubdate)
VALUES
('ZZ9999', '***No Current Title***',
'NONE', '9999', 0.00, 0, 0, 0, 'NONE', '1900.01.01')

-- Associate authors with no current titles to dummy title

INSERT titleauthor VALUES ('341-22-1782', 'ZZ9999', 0, 0)
INSERT titleauthor VALUES ('527-72-3246', 'ZZ9999', 0, 0)
INSERT titleauthor VALUES ('724-08-9931', 'ZZ9999', 0, 0)
INSERT titleauthor VALUES ('893-72-1158', 'ZZ9999', 0, 0)

-- Associate titles with no current author to dummy author

INSERT titleauthor VALUES ('000-00-0000', 'MC3026', 0, 0)

-- Now do a standard INNER JOIN

SELECT
'Author'=RTRIM(au_lname) + ', ' + au_fname,
'Title'=title
FROM
authors AS A -- JOIN conditions
INNER JOIN titleauthor AS TA ON A.au_id=TA.au_id
INNER JOIN titles AS T ON t.title_id=TA.title_id
WHERE
A.state <> 'CA'
ORDER BY 1

ROLLBACK TRAN -- Undo changes

This is the result:

Author Title
This document is created with the unregistered version of CHM2PDF Pilot

------------- ------------------
***No Current Author***, The Psychology of Computer Cooking
Blotchet-Halls, Reginald Fifty Years in Buckingham Palace
Kitchens
DeFrance, Michel The Gourmet Microwave
del Castillo, Innes Silicon Valley Gastronomic Treats
Greene, Morningstar ***No Current Title***
Panteley, Sylvia Onions, Leeks, and Garlic: Cooking
Secrets of the Mediterranean
Ringer, Albert Is Anger the Enemy?
Ringer, Albert Life Without Fear
Ringer, Anne The Gourmet Microwave
Ringer, Anne Is Anger the Enemy?
Smith, Meander ***No Current Title***

Earlier, you saw how OUTER JOIN became complicated and prone to mistakes. It's comforting to know that,
regardless of the order in which you specify the tables for the join operation using dummy rows, the semantics and
results are the same because now this operation is simply an INNER JOIN.

However, dummy values won't solve all the woes created by using NULL. First, using dummy values demands more
work in your application to ensure that relationships to the placeholder values are maintained. Even more important,
many queries still need to be able to discern that the placeholders are special values, or else the results of the query
might be suspect. For example, if you use the placeholder value of 0 (zero) instead of NULL for a salary field in an
employee table, you have to be careful not to use such a value in a query that looks for average values or minimum
salaries because a salary of 0 would alter your average or minimum calculations inappropriately. If the placeholder is
simply another designation for "unknown," you must write your queries with that in mind. Placeholder values usually
solve the issues that arise in joining tables, and they can be an effective alternative to using OUTER JOIN. With
placeholder values, developers—especially those with only basic SQL knowledge—are less likely to write incorrect
queries.

In the previous example, the naïve query returns the same result set as the query with an OUTER JOIN and the
criteria OR state IS NULL. It took careful work to construct the placeholder data and maintain the relationships for
this to be the case. But the group database expert (probably you!) can do that work, and once it is done, individuals
with less SQL or database knowledge won't be as likely to introduce bugs in their queries simply because they're
unfamiliar with the issues of OUTER JOINS and three-valued logic. Sometimes you might feel you have to use NULL
values; however, if you can minimize or avoid these cases, you're wise to do so.

If dummy entries aren't practical for you to use, try a few other tricks. A handy and underused function is ISNULL.
Using ISNULL(expression, value) substitutes the specified value for the expression when the expression evaluates to
NULL. In the earlier outer-join example, the WHERE clause was specified as
WHERE state <> 'CA' OR state IS NULL

By using the ISNULL function, you can assign a special value (in this case, XX) to any NULL state values. Then the
expression <> 'CA' will always evaluate to TRUE or FALSE rather than to Unknown. The clause OR state IS
NULL isn't needed.

The following query produces the full outer-join result, without using IS NULL:
SELECT
'Author'=RTRIM(au_lname) + ', ' + au_fname,
'Title'=title
FROM
authors AS A -- JOIN CONDITIONS
FULL OUTER JOIN titleauthor AS TA ON A.au_id=TA.au_id
FULL OUTER JOIN titles AS T ON t.title_id=TA.title_id
WHERE
This document is created with the unregistered version of CHM2PDF Pilot

ISNULL(state, 'XX') <> 'CA'

ORDER BY 1

IS NULL and = NULL

You've seen how the IS NULL operator exists to handle the special case of looking for NULL. Because NULL is
considered Unknown, ANSI-standard SQL doesn't provide for an equality expression of NULL—that is, there's no
WHERE value = NULL. Instead, you must use the IS NULL operator WHERE value IS NULL. This can make
queries more awkward to write and less intuitive to non-SQL experts. And it's somewhat curious, or at least
nonorthogonal, that ANSI SQL doesn't force an UPDATE statement to set a value TO NULL but instead uses the =
NULL formulation. (See Table 73 for a summary of this state of affairs.)
NOTE
Orthogonal is a somewhat jargony adjective that's often applied to programming languages and APIs. It's a
mathematical term meaning that the sum of the products of corresponding elements in a system equals 0. Thus, a
system is orthogonal if all of its primitive operations fully describe the base capabilities of the system and those
primitives are independent and mutually exclusive of one another. In a broader sense, the term denotes a certain
symmetry and consistency. For example, an API that has a GetData call also has a PutData call, each taking
arguments that are as consistent as possible between the two. The two APIs make up a mutually exclusive domain of
operations that you can perform on data, and they are named and invoked in a consistent manner. Such a pair of
APIs is considered orthogonal. SQL Server 2000 has a BULK INSERT command but no BULK EXPORT
command, so this is an example of nonorthogonal behavior.

Table 7-3. ANSI SQL's treatment of NULLs.

ANSI SQL Doesn't Provide for This And Provides Only for This
SELECT… WHERE salary = NULL SELECT? WHERE salary IS NULL
Yet ANSI SQL Uses Only This And Not Something Like This
UPDATE employee SET UPDATE employee SET salary TO
salary = NULL NULL

SQL Server provides an extension to ANSI SQL that allows queries to use = NULL as an alternative to IS NULL.
However, this alternative is available only when the ANSI_NULLS session option is OFF. When the ANSI_NULLS
option is ON, a condition testing for = NULL is always UNKNOWN.

Note that the formulation is not = 'NULL'— the quotation marks would have the effect of searching for a character
string containing the word NULL, as opposed to searching for the special meaning of NULL. The = NULL
formulation is simply a shorthand for IS NULL and in some cases provides a more convenient formulation of queries,
particularly by allowing NULL within an IN clause. The IN clause is a standard SQL shorthand notation for an
expression that checks multiple values, as illustrated here:
Multiple OR Conditions Equivalent Shorthand Formulation
WHERE state = 'CA' OR state = 'WA' OR state ='IL' WHERE state IN ('CA', 'WA', 'IL')

If you want to find states that are either one of the three values depicted in the previous example or NULL, ANSI
SQL prescribes that the condition be stated using both an IN clause and an OR state IS NULL expression; SQL
Server also allows all the conditions to be specified using the IN clause, as shown here:
ANSI SQL Prescribes SQL Server Also Allows
WHERE state IN ('CA', 'WA', 'IL') OR state IS NULL WHERE state IN ('CA', 'WA', 'IL', NULL)

The = NULL shorthand can be particularly handy when you're dealing with parameters passed to a stored procedure,
and this is the main reason it exists. Here's an example:
CREATE PROCEDURE get_employees (@dept char(8), @class char(5))
This document is created with the unregistered version of CHM2PDF Pilot

AS
SELECT * FROM employee WHERE employee.dept=@dept
AND employee.class=@class

By default, you can pass NULL to this procedure for either parameter; no special treatment is necessary. But without
the = NULL shorthand, you need to put in extensive special handling even for this simple procedure:
CREATE PROCEDURE get_employees (@dept char(8), @class char(5))
AS
IF (@dept IS NULL AND @class IS NOT NULL)
SELECT * FROM employee WHERE employee.dept IS NULL
AND employee.class=@class
ELSE IF (@dept IS NULL AND @class IS NULL)
SELECT * FROM employee WHERE employee.dept IS NULL
AND employee.class IS NULL
ELSE IF (@dept IS NOT NULL AND @class IS NULL)
SELECT * FROM employee WHERE employee.dept=@dept
AND employee.class IS NULL
ELSE
SELECT * FROM employee WHERE employee.dept=@dept
AND employee.class=@class

This example is pretty trivial; the situation becomes much more complex if you have multiple parameters and any one
of them might be NULL. You then need a SELECT statement for every combination of parameters that might be
NULL or NOT NULL. It can get ugly quickly.

Because your ODBC driver or OLE DB provider can enable the ANSI_NULLS option for you, in effect disallowing
the = NULL syntax, I recommend that you write your applications using only IS NULL. If you do have situations in
which you might pass NULLs as parameters to a procedure, you have another solution besides the multiple
SELECTs described above for every condition of NULL and NOT NULL parameter values. Stored procedures
keep track of what the ANSI_NULLS setting was at the time the procedure was created. You can open a Query
Analyzer session and explicitly SET ANSI_NULLS OFF. Then you can create all your procedures that might
otherwise require separate SELECTs to check for NULL parameters and NOT NULL parameters. Once created,
those procedures can be run in any connection, whether ANSI_NULLS is ON or OFF, and they will always be able
to compare a column to a parameter, whether the parameter is NULL or not.
This document is created with the unregistered version of CHM2PDF Pilot

34

Subqueries
SQL Server has an extremely powerful capability for nesting queries that provides a natural and efficient way to
express WHERE clause criteria in terms of the results of other queries. You can express most joins as subqueries,
although this method is often less efficient than performing a join operation. For example, to use the pubs database to
find all employees of the New Moon Books publishing company, you can write the query as either a join (using ANSI
join syntax) or as a subquery.

Here's the query as a join (equijoin, or inner join):

SELECT emp_id, lname
FROM employee JOIN publishers ON employee.pub_id=publishers.pub_id
WHERE pub_name='New Moon Books'

This is the query as a subquery:

SELECT emp_id, lname
FROM employee
WHERE employee.pub_id IN
(SELECT publishers.pub_id
FROM publishers WHERE pub_name='New Moon Books')

You can write a join (equijoin) as a subquery (subselect), but the converse isn't necessarily true. The equijoin offers
an advantage in that the two sides of the equation equal each other and the order doesn't matter. In certain types of
subqueries, it does matter which query is the nested query. However, if the query with the subquery can be rephrased
as a semantically equivalent JOIN query, the optimizer will do the conversion internally and the performance will be
the same whether you write your queries as joins or with subqueries.

Relatively complex operations are simple to perform when you use subqueries. For example, earlier you saw that the
pubs sample database has four rows in the authors table that have no related row in the titleauthor table (which
prompted our outer-join discussion). The following simple subquery returns those four author rows:
SELECT 'Author ID'=A.au_id,
'Author'=CONVERT(varchar(20), RTRIM(au_lname) + ', '
+ RTRIM(au_fname)), state
FROM authors A
WHERE A.au_id NOT IN
(SELECT B.au_id FROM titleauthor B)

Here's the output:

Author ID Author state
--------- ------------------- -----
341-22-1782 Smith, Meander KS
527-72-3246 Greene, Morningstar TN
724-08-9931 Stringer, Dirk CA
893-72-1158 McBadden, Heather CA

The IN operation is commonly used for subqueries, either to find matching values (similar to a join) or to find
nonmatching values by negating it (NOT IN), as shown above. Using the IN predicate is actually equivalent to =
This document is created with the unregistered version of CHM2PDF Pilot

ANY. If you wanted to find every row in authors that had at least one entry in the titleauthor table, you could use
either of these queries.

Here's the query using IN:

SELECT 'Author ID'=A.au_id,
'Author'=CONVERT(varchar(20), RTRIM(au_lname) + ', '
+ RTRIM(au_fname)), state
FROM authors A
WHERE A.au_id IN
(SELECT B.au_id FROM titleauthor B)

This is the query using equivalent formulation with = ANY:

SELECT 'Author ID'=A.au_id,
'Author'=CONVERT(varchar(20), RTRIM(au_lname) + ', '
+ RTRIM(au_fname)), state
FROM authors A
WHERE A.au_id=ANY
(SELECT B.au_id FROM titleauthor B)

Here's the output:

Author ID Author state
--------- -------------------- -----
172-32-1176 White, Johnson CA
213-46-8915 Green, Marjorie CA
238-95-7766 Carson, Cheryl CA
267-41-2394 O'Leary, Michael CA
274-80-9391 Straight, Dean CA
409-56-7008 Bennet, Abraham CA
427-17-2319 Dull, Ann CA
472-27-2349 Gringlesby, Burt CA
486-29-1786 Locksley, Charlene CA
648-92-1872 Blotchet-Halls, Regi OR
672-71-3249 Yokomoto, Akiko CA
712-45-1867 del Castillo, Innes MI
722-51-5454 DeFrance, Michel IN
724-80-9391 MacFeather, Stearns CA
756-30-7391 Karsen, Livia CA
807-91-6654 Panteley, Sylvia MD
846-92-7186 Hunter, Sheryl CA
899-46-2035 Ringer, Anne UT
998-72-3567 Ringer, Albert UT

Each of these formulations is equivalent to testing the value of au_id in the authors table to the au_id value in the first
row in the titleauthor table, and then OR'ing it to a test of the au_id value of the second row, and then OR'ing it to a
test of the value of the third row, and so on. As soon as one row evaluates to TRUE, the expression is TRUE, and
further checking can stop because the row in authors qualifies. However, it's an easy mistake to conclude that NOT
IN must be equivalent to <> ANY, and some otherwise good discussions of the SQL language have made this exact
mistake. More significantly, some products have also erroneously implemented it as such. Although IN is equivalent to
= ANY, NOT IN is instead equivalent to <> ALL, not to <> ANY.
NOTE
Careful reading of the ANSI SQL-92 specifications also reveals that NOT IN is equivalent to <> ALL but is not
equivalent to <> ANY. Section 8.4 of the specifications shows that R NOT IN T is equivalent to NOT (R = ANY
T). Furthermore, careful study of section 8.7 <quantified comparison predicate> reveals that NOT (R = ANY T) is
TRUE if and only if R <> ALL T is TRUE. In other words, NOT IN is equivalent to <> ALL.
This document is created with the unregistered version of CHM2PDF Pilot

By using NOT IN, you're stating that none of the corresponding values can match. In other words, all of the values
must not match (<> ALL), and if even one does match, it's FALSE. With <> ANY, as soon as one value is found to
be not equivalent, the expression is TRUE. This, of course, is also the case for every row of authors: rows in
titleauthor will always exist for other au_id values, and hence all authors rows will have at least one nonmatching
row in titleauthor. That is, every row in authors will evaluate to TRUE for a test of <> ANY row in titleauthor.

The following query using <> ALL returns the same four rows as the earlier one that used NOT IN:
SELECT 'Author ID'=A.au_id,
'Author'=CONVERT(varchar(20), RTRIM(au_lname) + ', '
+ RTRIM(au_fname)), state
FROM authors A
WHERE A.au_id <> ALL
(SELECT B.au_id FROM titleauthor B)

Here is the output:

Author ID Author state
--------- ------------------- -----
341-22-1782 Smith, Meander KS
527-72-3246 Greene, Morningstar TN
724-08-9931 Stringer, Dirk CA
893-72-1158 McBadden, Heather CA

If you had made the mistake of thinking that because IN is equivalent to = ANY, then NOT IN is equivalent to <>
ANY, you would have written the query as follows. This returns all 23 rows in the authors table!
SELECT 'Author ID'=A.au_id,
'Author'=CONVERT(varchar(20), RTRIM(au_lname) + ', '
+ RTRIM(au_fname)), state
FROM authors A
WHERE A.au_id <> ANY
(SELECT B.au_id FROM titleauthor B)

Here's the output:

Author ID Author state
--------- -------------------- -----
172-32-1176 White, Johnson CA
213-46-8915 Green, Marjorie CA
238-95-7766 Carson, Cheryl CA
267-41-2394 O'Leary, Michael CA
274-80-9391 Straight, Dean CA
341-22-1782 Smith, Meander KS
409-56-7008 Bennet, Abraham CA
427-17-2319 Dull, Ann CA
472-27-2349 Gringlesby, Burt CA
486-29-1786 Locksley, Charlene CA
527-72-3246 Greene, Morningstar TN
648-92-1872 Blotchet-Halls, Regi OR
672-71-3249 Yokomoto, Akiko CA
712-45-1867 del Castillo, Innes MI
722-51-5454 DeFrance, Michel IN
724-08-9931 Stringer, Dirk CA
724-80-9391 MacFeather, Stearns CA
756-30-7391 Karsen, Livia CA
807-91-6654 Panteley, Sylvia MD
846-92-7186 Hunter, Sheryl CA
This document is created with the unregistered version of CHM2PDF Pilot

893-72-1158 McBadden, Heather CA

899-46-2035 Ringer, Anne UT
998-72-3567 Ringer, Albert UT

The examples just shown use IN, NOT IN, ANY, and ALL to compare values to a set of values from a subquery.
This is common. However, it's also common to use expressions and compare a set of values to a single, scalar value.
For example, to find titles whose royalties exceed the average of all royalty values in the roysched table by 25
percent or more, you can use this simple query:
SELECT titles.title_id, title, royalty
FROM titles
WHERE titles.royalty >=
(SELECT 1.25 * AVG(roysched.royalty) FROM roysched)

This query is perfectly good because the aggregate function AVG (expression) stipulates that the subquery must
return exactly one value and no more. Without using IN, ANY, or ALL (or their negations), a subquery that returns
more than one row will result in an error. If you incorrectly rewrote the query as follows, without the AVG function,
you'd get run-time error 512:
SELECT titles.title_id, title, royalty
FROM titles
WHERE titles.royalty >=
(SELECT 1.25 * roysched.royalty FROM roysched)

Here is the output:

Server: Msg 512, Level 16, State 1, Line 1
Subquery returned more than 1 value. This is not permitted when the
subquery follows =, !=, <, <= , >, >= or when the subquery is used
as an expression.

It is significant that this error is a run-time error and not a syntax error: in the SQL Server implementation, if that
subquery didn't produce more than one row, the query would be considered valid and would execute. For example,
the subquery in the following code returns only one row, so the query is valid and returns four rows:
SELECT titles.title_id, royalty, title
FROM titles
WHERE titles.royalty >=
(SELECT 1.25*roysched.royalty FROM roysched
WHERE roysched.title_id='MC3021' AND lorange=0)

Here is the output:

title_id royalty title
-------- ----------- ---------------------------------------------
BU2075 24 You Can Combat Computer Stress!
MC3021 24 The Gourmet Microwave
PC1035 16 But Is It User Friendly?
TC4203 14 Fifty Years in Buckingham Palace Kitchens

However, this sort of query can be dangerous, and you should avoid it or use it only when you know that a
PRIMARY KEY or UNIQUE constraint will ensure that the subquery returns only one value. The query here
appears to work, but it's a bug waiting to happen. As soon as another row is added to the roysched table—say, with
a title_id of MC3021 and a lorange of 0—the query returns an error. No constraint exists to prevent such a row
This document is created with the unregistered version of CHM2PDF Pilot

from being added.

You might argue that SQL Server should determine whether a query formation could conceivably return more than
one row regardless of the data at the time and then disallow such a subquery formulation. The reason it doesn't is that
such a query might be quite valid when the database relationships are properly understood, so the power shouldn't be
limited to try to protect naïve users. Whether you agree with this philosophy or not, it's consistent with SQL in
general—and you should know by now that you can easily write a perfectly legal, syntactically correct query that
answers a question in a way that's entirely different from what you thought you were asking!

Correlated Subqueries
You can use powerful correlated subqueries to compare specific rows of one table to a condition in a matching table.
For each row otherwise qualifying in the main (or top) query, the subquery is evaluated. Conceptually, a correlated
subquery is similar to a loop in programming, although it's entirely without procedural constructs such as do-while or
for. The results of each execution of the subquery must be correlated to a row of the main query. In the next example,
for every row in the titles table that has a price of $19.99 or less, the row is compared with each sales row for stores
in California for which the revenue (price × qty) is greater than $250. In other words, "Show me titles with prices of
under $20 for which any single sale in California was more than $250."
SELECT T.title_id, title
FROM titles T
WHERE price <= 19.99
AND T.title_id IN (
SELECT S.title_id FROM sales S, stores ST
WHERE S.stor_id=ST.stor_id
AND ST.state='CA' AND S.qty*T.price > 250
AND T.title_id=S.title_id)

Here's the result:

title_id title
-------- -----------------------------
BU7832 Straight Talk About Computers
PS2091 Is Anger the Enemy?
TC7777 Sushi, Anyone?

Notice that this correlated subquery, like many subqueries, could have been written as a join (here using the old-style
JOIN syntax):
SELECT T.title_id, T.title
FROM sales S, stores ST, titles T
WHERE S.stor_id=ST.stor_id
AND T.title_id=S.title_id
AND ST.state='CA'
AND T.price <= 19.99
AND S.qty*T.price > 250

It becomes nearly impossible to create alternative joins when the subquery isn't doing a simple IN or when it uses
aggregate functions. For example, suppose we want to find titles that lag in sales for each store. This could be defined
as "Find any title for every store in which the title's sales in that store are below 80 percent of the average of sales for
all stores that carry that title and ignore titles that have no price established (that is, the price is NULL)." An intuitive
way to do this is to first think of the main query that will give us the gross sales for each title and store, and then for
each such result, do a subquery that finds the average gross sales for the title for all stores. Then we correlate the
subquery and the main query, keeping only rows that fall below the 80 percent standard.
This document is created with the unregistered version of CHM2PDF Pilot

Such an example follows. For clarity, notice the two distinct queries, each of which answers a separate question.
Then notice how they can be combined into a single correlated query to answer the specific question posed here. All
three queries use the old-style JOIN syntax.
-- This query computes gross revenues by
-- title for every title and store
SELECT T.title_id, S.stor_id, ST.stor_name, city, state,
T.price*S.qty
FROM titles AS T, sales AS S, stores AS ST
WHERE T.title_id=S.title_id AND S.stor_id=ST.stor_id

-- This query computes 80% of the average gross revenue for each
-- title for all stores carrying that title:
SELECT T2.title_id, .80*AVG(price*qty)
FROM titles AS T2, sales AS S2
WHERE T2.title_id=S2.title_id
GROUP BY T2.title_id

-- Correlated subquery that finds store-title combinations whose

-- revenues are less than 80% of the average of revenues for that
-- title for all stores selling that title
SELECT T.title_id, S.stor_id, ST.stor_name, city, state,
Revenue=T.price*S.qty
FROM titles AS T, sales AS S, stores AS ST
WHERE T.title_id=S.title_id AND S.stor_id=ST.stor_id
AND T.price*S.qty <
(SELECT 0.80*AVG(price*qty)
FROM titles T2, sales S2
WHERE T2.title_id=S2.title_id
AND T.title_id=T2.title_id )

And the answer is (from the third query):

title_id stor_id stor_name city state Revenue
-------- ------- ------------------- ---- ----- -------
BU1032 6380 Eric the Read Books Seattle WA 99.95
MC3021 8042 Bookbeat Portland OR 44.85
PS2091 6380 Eric the Read Books Seattle WA 32.85
PS2091 7067 News & Brews Los Gatos CA 109.50
PS2091 7131 Doc-U-Mat: Quality Remulade WA 219.00
Laundry and Books

When the newer ANSI JOIN syntax was first introduced, it wasn't obvious how to use it to write a correlated
subquery. It could be that the creators of the syntax forgot about the correlated subquery case, because using the
syntax seems like a hybrid of the old and the new: the correlation is still done in the WHERE clause rather than in the
JOIN clause. For illustration, examine the two equivalent formulations of the above query using the ANSI JOIN
syntax:
SELECT T.title_id, S.stor_id, ST.stor_name, city, state,
Revenue=T.price*S.qty
FROM titles AS T JOIN sales AS S ON T.title_id=S.title_id
JOIN stores AS ST ON S.stor_id=ST.stor_id
WHERE T.price*S.qty <
(SELECT 0.80*AVG(price*qty)
FROM titles T2 JOIN sales S2 ON T2.title_id=S2.title_id
WHERE T.title_id=T2.title_id )

SELECT T.title_id, S.stor_id, ST.stor_name, city, state,

Revenue=T.price*S.qty
FROM titles AS T JOIN sales AS S
ON T.title_id=S.title_id
This document is created with the unregistered version of CHM2PDF Pilot

AND T.price*S.qty <

(SELECT 0.80*AVG(T2.price*S2.qty)
FROM sales AS S2 JOIN titles AS T2
ON T2.title_id=S2.title_id
WHERE T.title_id=T2.title_id)
JOIN stores AS ST ON S.stor_id=ST.stor_id

To completely avoid the old-style syntax with the join condition in the WHERE clause, we could write this using a
subquery with a GROUP BY in the FROM clause (creating a derived table, which I'll describe in more detail later in
this section). However, although this gets around having to use the old syntax, it might not be worth it. The query is
much less intuitive than either of the preceding two formulations, and it takes twice as many logical reads to execute it.
SELECT T.title_id, S.stor_id, ST.stor_name, city, state,
Revenue = T.price * S.qty
FROM
titles AS T JOIN sales AS S
ON T.title_id = S.title_id
JOIN stores AS ST
ON S.stor_id = ST.stor_id
JOIN
(SELECT T2.title_id, .80 * AVG(price * qty) AS avg_val
FROM titles AS T2 JOIN sales AS S2
ON T2.title_id = S2.title_id
GROUP BY T2.title_id) AS AV ON T.title_id = AV.title_id
AND T.price * S.qty < avg_val

Often, correlated subqueries use the EXISTS statement, which is the most convenient syntax to use when multiple
fields of the main query are to be correlated to the subquery. (In practice, EXISTS is seldom used other than with
correlated subqueries.) EXISTS simply checks for a nonempty set. It returns (internally) either TRUE or NOT TRUE
(which we won't refer to as FALSE, given the issues of three-valued logic and NULL). Because no column value is
returned and the only thing that matters is whether any rows are returned, convention dictates that a column list isn't
specified. You can either use the * to indicate all columns or a constant such as the number 1 to indicate that the
column list is really ignored.

A common use for EXISTS is to answer a query such as "Show me the titles for which no stores have sales."
SELECT T.title_id, title FROM titles T
WHERE NOT EXISTS
(SELECT 1
FROM sales S
WHERE T.title_id=S.title_id )

Here is the output:

title_id title
-------- ----------------------------------
MC3026 The Psychology of Computer Cooking
PC9999 Net Etiquette

Conceptually, this query is pretty straightforward. The subquery, a simple equijoin, finds all matches of titles and
sales. Then NOT EXISTS correlates titles to those matches, looking for titles that don't have even a single row
returned in the subquery.

Another common use of EXISTS is to determine whether a table is empty. The optimizer knows that as soon as it
gets a single hit using EXISTS, the operation is TRUE and further processing is unnecessary. For example, here's how
you determine whether the authors table is empty:
This document is created with the unregistered version of CHM2PDF Pilot

SELECT 'Not Empty' WHERE EXISTS (SELECT * FROM authors)

Earlier, when discussing outer joins, I mentioned that you can now use an outer-join formulation to address what was
traditionally a problem in need of a correlated subquery solution. Here's an outer-join formulation for the problem
described earlier: "Show me the titles for which no stores have sales."
SELECT T1.title_id, title FROM titles T1
LEFT OUTER JOIN sales S ON T1.title_id=S.title_id
WHERE S.title_id IS NULL

TIP
Depending on your data and indexes, the outer-join formulation might be faster or slower than a correlated subquery.
But before deciding to write your query one way or the other, you might want to come up with a couple of alternative
formulations and then choose the one that's fastest in your situation.

In this example, for which little data exists, both solutions run in subsecond elapsed time. But the outer-join query
requires fewer than half the number of logical I/Os than does the correlated subquery. With more data, that difference
would be significant.

This query works by joining the stores and titles tables and by preserving the titles for which no store exists. Then, in
the WHERE clause, it specifically chooses only the rows that it preserved in the outer join. Those rows are the ones
for which a title had no matching store.

At other times, a correlated subquery might be preferable to a join, especially if it's a self-join back to the same table
or some other exotic join. Here's an example. Given the following table (and assuming that the row_num column is
guaranteed unique), suppose we want to identify the rows for which col2 and col3 are duplicates of another row:
row_num col2 col3
------- ---- ----
1 C D
2 A A
4 C B
5 C C
6 B C
7 C A
8 C B
9 C D
10 D D

We can do this in two standard ways. The first way uses a self-join. In a self-join, the table (or view) is used multiple
times in the FROM clause and is aliased at least once. Then it can be treated as an entirely different table and you can
compare columns between two "instances" of the same table. A self-join to find the rows having duplicate values for
col2 and col3 is easy to understand:
SELECT DISTINCT A.row_num, A.col2, A.col3
FROM match_cols AS A, match_cols AS B
WHERE A.col2=B.col2 AND A.col3=B.col3 AND A.row_num <> B.row_num
ORDER BY A.col2, A.col3

row_num col2 col3

------- ---- ----
4 C B
8 C B
1 C D
9 C D
This document is created with the unregistered version of CHM2PDF Pilot

But in this case, a correlated subquery using aggregate functions provides a considerably more efficient solution,
especially if many duplicates exist:
SELECT A.row_num, A.col2, A.col3 FROM match_cols AS A
WHERE EXISTS (SELECT B.col2, B.col3 FROM match_cols AS B
WHERE B.col2=A.col2
AND B.col3=A.col3
GROUP BY B.col2, B.col3 HAVING COUNT(*) > 1)
ORDER BY A.col2, A.col3

This correlated subquery has another advantage over the self-join example—the row_num column doesn't need to
be unique to solve the problem at hand.

You can take a correlated subquery a step further to ask a seemingly simple question that's surprisingly tricky to
answer in SQL: "Show me the stores that have sold every title." Even though it seems like a reasonable request,
relatively few people can come up with the correct SQL query, especially if I throw in the restrictions that you aren't
allowed to use an aggregate function like COUNT(*) and that the solution must be a single SELECT statement (that
is, you're not allowed to create temporary tables or the like).

The previous query already revealed two titles that no store has sold, so we know that with the existing dataset, no
stores can have sales for all titles. For illustrative purposes, let's add sales records for a hypothetical store that does, in
fact, have sales for every title. Following that, we'll see the query that finds all stores that have sold every title (which
we know ahead of time is only the phony one we're entering here):
-- The phony store
INSERT stores (stor_id, stor_name, stor_address, city, state, zip)
VALUES ('9999', 'WE SUPPLY IT ALL', 'One Main St', 'Poulsbo',
'WA', '98370')

-- By using a combination of hard-coded values and selecting every

-- title, generate a sales row for every title
INSERT sales (stor_id, title_id, ord_num, ord_date, qty, payterms)
SELECT '9999', title_id, 'PHONY1', GETDATE(), 10, 'Net 60'
FROM titles

-- Find stores that supply every title

SELECT ST.stor_id, ST.stor_name, ST.city, ST.state
FROM stores ST
WHERE NOT EXISTS
(SELECT * FROM titles T1
WHERE NOT EXISTS
(SELECT * FROM titles T1
WHERE NOT EXISTS
(SELECT * FROM sales S
WHERE S.title_id=T1.title_id AND ST.stor_id=S.stor_id)
)

Here's the result:

stor_id stor_name city state
------- --------- ---- -----
9999 WE SUPPLY IT ALL Poulsbo WA

Although this query might be difficult to think of immediately, you can easily understand why it works. In English, it
says, "Show me the store(s) such that no titles exist that the store doesn't sell." This query consists of the two
subqueries that are applied to each store. The bottommost subquery produces all the titles that the store has sold. The
This document is created with the unregistered version of CHM2PDF Pilot

upper subquery is then correlated to that bottom one to look for any titles that are not in the list of those that the store
has sold. The top query returns any stores that aren't in this list. This type of query is known as a relational division,
and unfortunately, it isn't as easy to express as we'd like. Although the query shown is quite understandable, once you
have a solid foundation in SQL, it's hardly intuitive. As is almost always the case, you could probably use other
formulations to write this query.

I've already alluded to writing this query without using an aggregate function like COUNT. There's nothing wrong
with using an aggregate function—I imposed this restriction only to make writing the query more of a challenge. If you
think of the query in English as "Find the stores that have sold as many unique titles as there are total unique titles,"
you'll find the following formulation somewhat more intuitive:
SELECT ST.stor_id, ST.stor_name
FROM stores ST, sales SA, titles T
WHERE SA.stor_id=ST.stor_id AND SA.title_id=T.title_id
GROUP BY ST.stor_id,ST.stor_name
HAVING COUNT(DISTINCT SA.title_id)=(SELECT COUNT(*) FROM titles T1)

The following formulation runs much more efficiently than either of the previous two. The syntax is similar to the
preceding one but its approach is novel because it's just a standard subquery, not a join or a correlated subquery.
You might think it's an illegal query, since it does a GROUP BY and a HAVING without an aggregate in the select list
of the first subquery. But that's OK, both in terms of what SQL Server allows and in terms of the ANSI specification.
What isn't allowed is having an item in the select list that isn't an aggregate function but then omitting it from the
GROUP BY clause, if there is a GROUP BY clause.
-- Find stores that have sold every title
SELECT stor_id, stor_name FROM stores WHERE stores.stor_id IN
(SELECT stor_id FROM sales GROUP BY stor_id
HAVING COUNT(DISTINCT title_id)=(SELECT COUNT(*) FROM titles)
)

And as a lead-in to the next topic, here's a formulation that uses a derived table—a feature that allows you to use a
subquery in a FROM clause. This capability lets you alias a virtual table returned as the result set of a SELECT
statement, and then lets you use this result set as if it were a real table. This query also runs efficiently.
SELECT ST.stor_id, ST.stor_name
FROM stores ST,
(SELECT stor_id, COUNT(DISTINCT title_id) AS title_count
FROM sales
GROUP BY stor_id
) as SA
WHERE ST.stor_id=SA.stor_id AND SA.title_count=
(SELECT COUNT(*) FROM titles)
This document is created with the unregistered version of CHM2PDF Pilot

34

Views and Derived Tables

Think of a view as a virtual table. Simply put, a view is a named SELECT statement that dynamically produces a
result set that you can further operate on. A view doesn't actually store any data. It acts as a filter for underlying tables
in which the data is stored. The SELECT statement that defines the view can be from one or more underlying tables
or from other views. To relieve users of the complexity of having to know how to write an outer join properly, we can
turn the previous outer-join query into a view:
CREATE VIEW outer_view AS
(
SELECT
'Author'=RTRIM(au_lname) + ', ' + au_fname, 'Title'=title
FROM (titleauthor AS TA
FULL OUTER JOIN titles AS T ON (TA.title_id=T.title_id))
RIGHT OUTER JOIN authors AS A ON (A.au_id=TA.au_id)
WHERE
A.state <> 'CA' OR A.state IS NULL
)

Now, instead of formulating the outer join, we can simply query the outer-join view, outer_view. Then we can do a
search for author names starting with Ri.
SELECT * FROM outer_view WHERE Author LIKE 'Ri%' ORDER BY Author

Here's the output:

Author Title
-------------- ---------------------
Ringer, Albert Is Anger the Enemy?
Ringer, Albert Life Without Fear
Ringer, Anne The Gourmet Microwave
Ringer, Anne Is Anger the Enemy?

Notice that the view defines the set of rows that are included, but it doesn't define their order. If we want a specific
order, we must specify the ORDER BY clause in the SELECT statement on the view rather than on the SELECT
statement that defines the view.

A derived table is a fancy name for the result of using another SELECT statement in the FROM clause of a
SELECT statement. This works because the result of a SELECT statement is a table, which is exactly what the
FROM clause requires. You can think of a view as a named derived table. A view is named, and its definition is
persistent and reusable; a derived table is a completely dynamic, temporal concept. To show the difference, here's an
equivalent LIKE 'Ri%' query using a derived table instead of a view:
SELECT *
FROM
(SELECT
'Author'=RTRIM(au_lname) + ',' + au_fname, 'Title'=title
FROM (titleauthor AS TA
FULL OUTER JOIN titles AS T ON (TA.title_id=T.title_id))
RIGHT OUTER JOIN authors AS A ON (A.au_id=TA.au_id)
WHERE A.state <> 'CA' OR A.state IS NULL
) AS T
This document is created with the unregistered version of CHM2PDF Pilot

WHERE T.Author LIKE 'Ri%'

You can insert, update, and delete rows in a view but not in a derived table. Keep in mind that you're always
modifying rows in the tables on which a view is based, because the view has no data of its own. The definition of the
view is persistent, but the data is not. The data exists only in the view's base tables. Think of it as modifying data
through a view rather than modifying data in a view. Some limitations to modifying data through a view exist; the
SQL Server documentation explains them, but they bear repeating here with a bit more comment.

The ability to modify data through a view is determined by the characteristics of the view, the properties of the base
tables, and the actual change being made. If the view has INSTEAD-OF triggers, then none of the following rules for
modification apply because the INSTEAD-OF trigger replaces the modification operations with other actions. (I'll
discuss INSTEAD-OF triggers in Chapter 12.) If the view is a partitioned view, it has its own rules for modification
(as discussed in the next section). If the view does not have relevant INSTEAD-OF triggers and is not a partitioned
view, it is considered updateable if it does not contain TOP, GROUP BY, UNION, or DISTINCT. You can use
aggregate functions only in a subquery in the view. So if the view is considered updateable by these rules, the
following restrictions apply to the actual modification statements:

Modifications restricted to one base table Data modification statements (INSERT and UPDATE only) are
allowed on multiple-table views if the data modification statement affects only one base table. DELETE statements are
never allowed on multiple-table views.

INSERT statements and NOT NULL columns INSERT statements aren't accepted unless all the NOT NULL
columns without defaults in the underlying table or view are included in the view through which you're inserting new
rows and values for those columns are included in the INSERT statement. SQL Server has no way to supply values
for NOT NULL columns in the underlying table or view if no default value has been defined.

Data restrictions All columns being modified must adhere to all restrictions for the data modification statement as if
they were executed directly against the base table. This applies to column nullability, constraints, identity columns, and
columns with rules and/or defaults and base table triggers.

Limitations on INSERT and UPDATE statements INSERT and UPDATE statements can't add or change any
column in a view that's a computation or the result of an aggregate function.

READTEXT or WRITETEXT You can't use READTEXT or WRITETEXT on text or image columns in views.

Modifications and view criteria By default, data modification statements through views aren't checked to
determine whether the rows affected are within the scope of the view. For example, you can issue an INSERT
statement for a view that adds a row to the underlying base table, but that doesn't add the row to the view. This
behavior occurs because the column values are all valid to the table, so they can be added; however, if the column
values don't meet the view's criteria, they aren't represented in the selection for the view. Similarly, you can issue an
UPDATE statement that changes a row in such a way that the row no longer meets the criteria for the view. If you
want all modifications to be checked, use the WITH CHECK OPTION option when you create the view.

For the most part, these restrictions are logical and understandable, except for WITH CHECK OPTION. For
instance, if you use a view that defines some select criteria and allows the rows produced by the view to be modified,
you must use WITH CHECK OPTION or you might experience the bizarre results of inserting a row that you can
never see or updating rows in such a way that they disappear from the view.

Here's an example:
CREATE VIEW CA_authors AS
(
SELECT * FROM authors
WHERE state='CA'
)
This document is created with the unregistered version of CHM2PDF Pilot

GO

SELECT * FROM CA_authors

-- (returns 15 rows)
BEGIN TRAN
UPDATE CA_authors SET state='IL'
SELECT * FROM CA_authors

-- (returns 0 rows)

ROLLBACK TRAN

Should you be able to update rows in a way that causes them to disappear from the view or add rows and never see
them again? It's a matter of opinion. But because the SQL Server behavior is as ANSI SQL specifies, and such
disappearing rows are as specified, you should consider always using WITH CHECK OPTION if a view is to be
updated and the criteria are such that WITH CHECK OPTION might be violated.

If we rewrite the view as follows

CREATE VIEW CA_authors AS
(
SELECT * FROM authors
WHERE state='CA'
)
WITH CHECK OPTION

when we attempt to update the state column to IL, the command fails and we get this error message:
Server: Msg 550, Level 16, State 1, Line 1
The attempted insert or update failed because the target view either
specifies WITH CHECK OPTION or spans a view that specifies WITH CHECK
OPTION and one or more rows resulting from the operation did not
qualify under the CHECK OPTION constraint.
The statement has been terminated.

Views also have an often-overlooked ability to use a system function in the view definition, making it dynamic for the
connection. Suppose we keep a personnel table that has a column called sqldb_name, which is the person's login
name to SQL Server. The function SUSER_SNAME (server_user_id) returns the login name for the current
connection. We can create a view using the SUSER_SNAME system function to limit that user to only his or her
specific row in the table:
CREATE VIEW My_personnel AS
(
SELECT * FROM personnel WHERE sqldb_name = SUSER_SNAME()
)
WITH CHECK OPTION

SQL Server provides system functions that return the application name, database username (which might be different
from the server login name), and workstation name, and you can use these functions in similar ways. You can see
from the previous example that a view can also be important in dealing with security. For details on the system
functions, see the SQL Server documentation.

Altering Views
This document is created with the unregistered version of CHM2PDF Pilot

SQL Server 2000 allows you to alter the definition of a view. The syntax is almost identical to the syntax for creating
the view initially:

ALTER VIEW view_name [(column [,?n])]

[WITH ENCRYPTION]
AS
select_statement
[WITH CHECK OPTION]

The big difference is that view_name must already exist and the definition specified replaces whatever definition the
view had before ALTER VIEW was executed. You might wonder why this is a useful command, since the entire
definition must be reentered in the ALTER VIEW. Why don't we just drop the view and re-create it? The benefit
comes from the fact that the view's object_id won't change, which means that all the internal references to this view
remain intact. If other views reference this one, they'll be unaffected. If you've assigned permissions on this view to
various users and roles, dropping the view means that all permission information is lost. Altering the view keeps the
permissions intact.

There is one gotcha when you deal with permissions on views if the permissions have been assigned on individual
columns. Internally, SQL Server keeps track of permissions by the column ID number. If you alter a view and change
the order of the columns, the permissions might no longer be what you expect. Suppose you've created a view on the
authors table to select the authors' names and phone numbers. You want all users to be able to see the authors'
names, but you want the phone numbers to be unavailable.
CREATE VIEW authorlist (first, last, phone)
AS
SELECT au_fname, au_lname, phone
FROM authors
GO
GRANT SELECT ON authorlist (first, last) TO PUBLIC
GO

All users in the pubs database can now execute this command:
SELECT first, last FROM authorlist

You can then alter the view to put the phone number column in the first position:
ALTER VIEW authorlist (phone, first, last)
AS
SELECT phone, au_fname, au_lname
FROM authors

Because the permissions have been granted on the first two columns, users will get an error message when they run
this query:
Server: Msg 230, Level 14, State 1
SELECT permission denied on column 'last' of object 'authorlist',
database 'pubs', owner 'dbo'.

However, all users can now select the values in the phone column, which you're trying to keep hidden.

Partitioned Views
This document is created with the unregistered version of CHM2PDF Pilot

Views in SQL Server can contain the UNION ALL clause, and if these UNION views meet certain conditions, they
are called partitioned views. Partitioned views let you separate disjoint sets of similar data into separate tables, but
through the view, all the tables can be accessed simultaneously. For example, if you were to separate your sales data
by quarter so that each quarter was in its own table, you could create a view for the year's sales that unioned the data
from all the quarterly tables. Alternatively, you could have sales data for each region of the country in a separate table
and use UNION ALL to combine all the regional tables into a view for the whole country.

A view can be considered a partitioned view if it meets all the following criteria:
 The columns in the corresponding positions in each SELECT list are of the same exact type, including having
the same collation.
 One column in each table, in the same position, must have a CHECK constraint defined on it that is mutually
exclusive with the corresponding constraints in the other tables. This column is called the partitioning column.
For example, one table could have the constraint (region = 'East'), one table could have the constraint (region
= 'North'), one could have (region = 'West'), and another could have (region = 'South'). No row could satisfy
more than one of these constraints.
 The partitioning column cannot be a computed column.
 If any of the underlying tables exists on servers other than the local SQL Server on which the view is defined,
the view is called a distributed partitioned view and is subject to additional restrictions. You can find out
more about distributed partitioned views in SQL Server Books Online.
 The same underlying table cannot appear more than once in the view.
 The underlying tables cannot have indexes on computed columns.

Here's an example of a partitioned view:

-- Create the tables and insert the values
CREATE TABLE Sales_West (
Ordernum INT,
total money,
region char(5) check (region = 'West'),
primary key (Ordernum, region)
)
CREATE TABLE Sales_North (
Ordernum INT,
total money,
region char(5) check (region = 'North'),
primary key (Ordernum, region)
)
CREATE TABLE Sales_East (
Ordernum INT,
total money,
region char(5) check (region = 'East'),
primary key (Ordernum, region)
)
CREATE TABLE Sales_South (
Ordernum INT,
total money,
region char(5) check (region = 'South'),
primary key (Ordernum, region)
)
GO

INSERT Sales_West VALUES (16544, 2465, 'West')

INSERT Sales_West VALUES (32123, 4309, 'West')
INSERT Sales_North VALUES (16544, 3229, 'North')
INSERT Sales_North VALUES (26544, 4000, 'North')
INSERT Sales_East VALUES ( 22222, 43332, 'East')
INSERT Sales_East VALUES ( 77777, 10301, 'East')
INSERT Sales_South VALUES (23456, 4320, 'South')
INSERT Sales_South VALUES (16544, 9999, 'South')
GO
This document is created with the unregistered version of CHM2PDF Pilot

-- Create the view that combines all sales tables

CREATE VIEW Sales_National
AS
SELECT *
FROM Sales_West
UNION ALL
SELECT *
FROM Sales_North
UNION ALL
SELECT *
FROM Sales_East
UNION ALL
SELECT *
FROM Sales_South

 Partitioned views are updateable under certain conditions. In addition to meeting all the above criteria for the
view to be partitioned, the view must meet these restrictions: the primary key column of each table should be
included in the select list and must include the partitioning column.
 All columns in the underlying tables not included in the SELECT list should allow NULLs. Default constraints
cannot be defined on these columns.

To INSERT into a partitioned view, you must supply a value for all non-nullable columns in the view, even if the
column has a default defined in the underlying table. In addition, the value being inserted into the partitioning column
must satisfy one of the CHECK constraints in one of the underlying tables; otherwise, you'll get an error message
stating that a constraint violation has occurred.

The SQL Server optimizer can be quite clever about processing queries that access partitioned views. In particular, it
can determine that it does not need to look at every table in the view, but only the table that contains the rows that
match the value of the partitioning column. Here's an example:
SELECT *
FROM Sales_National
WHERE region = 'South'

For this query, the optimizer can determine that the only table that needs to be accessed is the Sales_South table.
You can see that this is the case by inspecting the query plan produced when you execute this query. We'll look at the
details of examining query plans in Chapter 16.
This document is created with the unregistered version of CHM2PDF Pilot

34

Other Search Expressions

In addition to the SELECT statement and joins, SQL Server provides other useful search expressions that enable
you to tailor your queries. These expressions include LIKE and BETWEEN, along with aggregate functions such as
AVG, COUNT, and MAX.

LIKE
In an earlier example, we used LIKE to find authors whose names start with Ri. LIKE allows you to search using the
simple pattern matching that is standard with ANSI SQL, but SQL Server adds capability by introducing wildcard
characters that allow the use of regular expressions. Table 7-4 shows how wildcard characters are used in searches.

Table 7-4. Wildcard characters recognized by SQL Server for use in regular expressions.
Wildcard Searches for
% Any string of zero or more characters.
_ Any single character.
Any single character within the specified range (for
[] example, [a-f]) or the specified set (for example,
[abcdef]).
Any single character not within the specified range (for
[^] example, [^a-f]) or the specified set (for example,
[^abcdef]).

Table 7-5 shows examples from the SQL Server documentation of how LIKE can be used.

Table 7-5. Using LIKE in searches.

Form Searches for
All names that begin with the letters Mc (such as
LIKE 'Mc?%'
McBadden).
All names that end with inger (such as Ringer and
LIKE '?%inger'
Stringer).
All names that include the letters en (such as Bennet,
LIKE '?%en?%'
Green, and McBadden).
All six-letter names ending with heryl (such as Cheryl and
LIKE '_heryl'
Sheryl).
All names that begin with C or K, then ars, then e or o,
LIKE '[CK]ars[eo]n' and end with n (such as Carsen, Karsen, Carson, and
Karson).
All names ending with inger that begin with any single
LIKE '[M-Z]inger'
letter from M through Z (such as Ringer).
All names beginning with the letter M that don't have the
LIKE 'M[^c]%'
letter c as the second letter (such as MacFeather).

Trailing Blanks
Trailing blanks (blank spaces) can cause considerable confusion and errors. In SQL Server, when you use trailing
This document is created with the unregistered version of CHM2PDF Pilot

blanks you must be aware of differences in how varchar and char data is stored and of how the SET
ANSI_PADDING option is used.

Suppose that I create a table with five columns and insert data as follows:
SET ANSI_PADDING OFF

DROP TABLE checkpad

GO

CREATE TABLE checkpad

(
rowid smallint NOT NULL PRIMARY KEY,
c10not char(10) NOT NULL,
c10nul char(10) NULL,
v10not varchar(10) NOT NULL,
v10nul varchar(10) NULL
)

-- Row 1 has names with no trailing blanks

INSERT checkpad VALUES (1, 'John', 'John', 'John', 'John')

-- Row 2 has each name inserted with three trailing blanks

INSERT checkpad VALUES
(2, 'John ', 'John ', 'John ', 'John ')

-- Row 3 has each name inserted with a full six trailing blanks
INSERT checkpad VALUES
(3, 'John ', 'John ', 'John ', 'John ')

-- Row 4 has each name inserted with seven trailing blanks (too many)
INSERT checkpad VALUES
(4, 'John ', 'John ', 'John ', 'John ')

I can then use the following query to analyze the contents of the table. I use the DATALENGTH(expression)
function to determine the actual length of the data and then convert to VARBINARY to display the hexadecimal
values of the characters stored. (For those of you who don't have the ASCII chart memorized, a blank space is
0x20.) I then use a LIKE to try to match each column, once with one trailing blank and once with six trailing blanks.
SELECT ROWID,
"0xC10NOT"=CONVERT(VARBINARY(10), c10not), L1=DATALENGTH(c10not),
"LIKE 'John %'"=CASE WHEN (c10not LIKE 'John %') THEN 1 ELSE 0 END,
"LIKE 'John %'"=CASE WHEN (c10not LIKE 'John %')
THEN 1 ELSE 0 END,
"0xC10NUL"=CONVERT(VARBINARY(10), c10nul), L2=DATALENGTH(c10nul),
"LIKE 'John %'"=CASE WHEN (c10nul LIKE 'John %') THEN 1 ELSE 0 END,
"LIKE 'John %'"=CASE WHEN (c10nul LIKE 'John %')
THEN 1 ELSE 0 END,
"0xV10NOT"=CONVERT(VARBINARY(10), v10not), L3=DATALENGTH(v10not),
"LIKE 'John %'"=CASE WHEN (v10not LIKE 'John %') THEN 1 ELSE 0 END,
"LIKE 'John %'"=CASE WHEN (v10not LIKE 'John %')
THEN 1 ELSE 0 END,
"0xV10NUL"=CONVERT(VARBINARY(10), v10nul), L4=DATALENGTH(v10nul),
"LIKE 'John %'"=CASE WHEN (v10nul LIKE 'John %') THEN 1 ELSE 0 END,
"LIKE 'John %'"=CASE WHEN (v10nul LIKE 'John %')
THEN 1 ELSE 0 END
FROM checkpad

Figure 7-2 shows the results of this query from the checkpad table.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 7-2. The results of the query using SET ANSI_PADDING OFF.

If you don't enable SET ANSI_PADDING ON, SQL Server trims trailing blanks for variable-length columns and
for fixed-length columns that allow NULLs. Notice that only column c10not has a length of 10. The other three
columns are treated identically and end up with a length of 4 and no padding. For these columns, even trailing blanks
explicitly entered within the quotation marks are trimmed. All the variable-length columns as well as the fixed-length
column that allows NULLs have only the 4 bytes J-O-H-N (4A-6F-68-6E) stored. The true fixed-length column,
c10not, is automatically padded with six blank spaces (0x20) to fill the entire 10 bytes.

All four columns match LIKE 'John%' (no trailing blanks), but only the fixed-length column c10not matches LIKE
'John ?' (one trailing blank) and LIKE 'JOHN %'?' (six trailing blanks). The reason is apparent when you realize that
those trailing blanks are truncated and aren't part of the stored values for the other columns. A column query gets a hit
with LIKE and trailing blanks only when the trailing blanks are stored. In this example, only the c10not column stores
trailing blanks, so that's where the hit occurs.

SQL Server has dealt with trailing blanks in this way for a long time. However, when the developers began working
to pass the National Institute of Standards and Technology (NIST) test suite for ANSI SQL conformance, they
realized that this behavior was inconsistent with the ANSI specification. The specification requires that you always
pad char datatypes and that you don't truncate trailing blanks entered by the user, even for varchar. I could argue for
either side—both the long-standing SQL Server behavior and the ANSI specification behavior can be desirable,
depending on the circumstances. However, if SQL Server had changed its behavior to conform to ANSI, it would
have broken many deployed SQL Server applications. So, to satisfy both sides, the ANSI_PADDING option was
added in version 6.5. If you enable SET ANSI_PADDING ON, re-create the table, reinsert the rows, and then issue
the same query, you get the results shown in Figure 7-3, which are different from those shown in Figure 7-2.

If you set ANSI_PADDING ON, the char(10) columns c10not and c10nul, whether declared NOT NULL or
NULL, are always padded out to 10 characters regardless of how many trailing blanks were entered. In addition, the
variable-length columns v10not and v10nul contain the number of trailing blanks that were inserted. Trailing blanks
are not truncated. Because the variable-length columns now store any trailing blanks that were part of the INSERT
(or UPDATE) statement, the columns get hits in queries with LIKE and trailing blanks if the query finds that number
of trailing blanks (or more) in the columns' contents. Notice that no error occurs as a result of inserting a column with
seven (instead of six) trailing blanks. The final trailing blank simply gets truncated. If you want a warning message to
appear in such cases, you can use SET ANSI_WARNINGS ON.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 7-3. The results of the query using SET ANSI_PADDING ON.

Note that the SET ANSI_PADDING ON option applies to a table only if the option was enabled at the time the
table was created. (SET ANSI_PADDING ON also applies to a column if the option was enabled at the time the
column was added via ALTER TABLE.) Enabling this option for an existing table does nothing. To determine
whether a column has ANSI_PADDING set to ON, examine the status field of the syscolumns table and do a
bitwise AND 20 (decimal). If the bit is on, you know the column was created with the option enabled. For example:
SELECT name, 'Padding On'=CASE WHEN status & 20 <> 0
THEN 'YES' ELSE 'NO' END
FROM syscolumns WHERE id=OBJECT_ID('checkpad') AND name='v10not'
GO

SET ANSI_PADDING ON also affects the behavior of binary and varbinary fields, but the padding is done with
zeros instead of spaces. Table 7-6 shows the effects of the SET ANSI_PADDING setting when values are inserted
into columns with char, varchar, binary, and varbinary data types.

Note that when a fixed-length column allows NULLs, it mimics the behavior of variable-length columns if
ANSI_PADDING is off. If ANSI_PADDING is on, fixed-length columns that allow NULLs behave the same as
fixed-length columns that don't allow NULLs.

The fact that a fixed-length character column that allows NULLs is treated as a variable-length column does not
affect how the data is actually stored on the page. If you use the DBCC PAGE command (as discussed in Chapter 6)
to look at the column c10nul, you'll see that it always holds a full 10 characters, including the 6 trailing spaces in our
example above. When you manipulate the column using SQL statements, the query processor checks the status of the
bit in syscolumns.status, and if the column is marked with ANSI_PADDING set to ON, the query processor
ignores trailing blanks.

You should also be aware that comparing these strings using LIKE is quite a bit different than comparing the strings
for equality. The following query returns only rows in which the query processor recognizes at least one trailing space
after the 'John':
SELECT * FROM checkpad
WHERE c10nul LIKE 'John %'

On the other hand, a comparison that looks for equality always ignores any trailing spaces and finds a match
whenever all the leading, nonspace characters are the same. The following query returns all the rows in the checkpad
table, whether ANSI_PADDING is ON or OFF, and in addition, it returns all the rows in the table if you replace the
c10nul column with any of the columns in the table:
SELECT * from checkpad
WHERE c10nul = 'John'
This document is created with the unregistered version of CHM2PDF Pilot

The SQL Server ODBC driver and OLE DB Provider for SQL Server automatically set ANSI_PADDING to ON
when they connect. When you write your own applications, the ANSI_PADDING setting can be changed in the
ODBC connection attributes or the OLE DB connection properties before your application makes its connection to
the SQL Server. In addition, the ANSI_PADDING setting can be changed in the ODBC Data Sources dialog box,
available through the Control Panel.

Table 7-6. The effects of the ANSI_PADDING setting.

Setting of char(n) NOT NULL or char(n) NULL or varchar(n) or
ANSI_PADDING binary(n) NOT NULL binary(n) NULL varbinary(n)
Trailing blanks (for varchar)
or zeros (for varbinary) in
Follows same rules as for
Pad original value to the values inserted aren't
ON char(n) or binary(n) NOT
length of the column. trimmed. Values are not
NULL.
padded to the length of the
column.
Trailing blanks (for varchar)
Pad original value to the Follows same rules as for
OFF or trailing zeros (for
length of the column. varchar or varbinary.
varbinary) are trimmed.

Regardless of how the column was created, SQL Server provides functions that can help you deal effectively with
trailing blanks. The RTRIM(char_expression) function removes trailing blanks if they exist. You can add trailing
blanks to your output by concatenating with the SPACE(integer_expression) function. An easier way is to use the
CONVERT function to convert to a fixed-length char of whatever length you want. Then variable-length columns will
be returned as fully padded, regardless of the ANSI_PADDING setting and regardless of the columns' actual
storage, by converting them to char(10) or whatever size you want. The following SELECT statement against the
checkpad table illustrates this:
SELECT CONVERT(VARBINARY(10), CONVERT(char(10), v10nul))
FROM checkpad

Here's the result:

0x4a6f686e202020202020
0x4a6f686e202020202020
0x4a6f686e202020202020
0x4a6f686e202020202020

BETWEEN
BETWEEN is shorthand for greater-than-or-equal-to AND less-than-or-equal-to. The clause WHERE C
BETWEEN B AND D is equivalent to WHERE C >= B AND C <= D. The order of the values to be checked is
important when you use BETWEEN. If B, C, and D are increasing values, for example, WHERE C BETWEEN D
AND B isn't true because it evaluates to WHERE C >= D AND C <= B.

Aggregate Functions
Aggregate functions (sometimes referred to as set functions) allow you to summarize a column of output. SQL
Server provides six general aggregate functions that are standard ANSI SQL-92 fare. I won't go into much detail
about their general use; instead, I'll focus on some aspects specific to SQL Server. Table 7-7 summarizes SQL
Server's ANSI SQL aggregate functions.
This document is created with the unregistered version of CHM2PDF Pilot

Consider the following table, automobile_sales_detail. I'll use this table to demonstrate how aggregate functions
simplify finding the answers to complex questions:
rowid model year color units_sold
----- ----- ---- ----- ----------
1 Chevy 1990 Red 5
2 Chevy 1990 White 87
3 Chevy 1990 Blue 62
4 Chevy 1991 Red 54
5 Chevy 1991 White 95
6 Chevy 1991 Blue 49
7 Chevy 1992 Red 31
8 Chevy 1992 White 54
9 Chevy 1992 Blue 71
10 Ford 1990 Red 64
11 Ford 1990 White 62
12 Ford 1990 Blue 63
13 Ford 1991 Red 52
14 Ford 1991 White 9
15 Ford 1991 Blue 55
16 Ford 1992 Red 27
17 Ford 1992 White 62
18 Ford 1992 Blue 39

Table 7-7. SQL Server's ANSI aggregate functions.

Aggregate Function Description
Returns the average (mean) of all the values, or only the
AVG(expression) DISTINCT values, in the expression. You can use AVG
with numeric columns only.* Null values are ignored.
Returns the number of non-null values in the expression.
When DISTINCT is specified, COUNT finds the number
COUNT(expression) of unique non-null values. You can use COUNT with
both numeric and character columns. Null values are
ignored.
Returns the number of rows. COUNT(*) takes no
COUNT(*) parameters and can't be used with DISTINCT. All rows
are counted, even those with null values.
Returns the maximum value in the expression. You can
use MAX with numeric, character, and datetime columns
but not with bit columns. With character columns, MAX
MAX(expression)
finds the highest value in the collating sequence. MAX
ignores any null values. DISTINCT is available for ANSI
compatibility, but it's not meaningful with MAX.
Returns the minimum value in the expression. You can use
MIN with numeric, character, and datetime columns, but
not with bit columns. With character columns, MIN finds
MIN(expression)
the value that is lowest in the sort sequence. MIN ignores
any null values. DISTINCT is available for ANSI
compatibility, but it's not meaningful with MIN.
Returns the sum of all the values, or only the DISTINCT
SUM(expression) values, in the expression. You can use SUM with numeric
columns only. Null values are ignored.

* As used here, numeric columns refers to decimal, float, int, money, numeric, real, smallint, smallmoney, bigint, and
tinyint datatypes.
This document is created with the unregistered version of CHM2PDF Pilot

You can increase the power of aggregate functions by allowing them to be grouped and by allowing the groups to
have criteria established for inclusion via the HAVING clause. The following queries provide some examples.

EXAMPLE 1

Show the oldest and newest model years for sale, and show the average sales of all the entries in the table:
SELECT 'Oldest'=MIN(year), 'Newest'=MAX(year),
'Avg Sales'=AVG(units_sold)
FROM automobile_sales_detail

Here's the output:

Oldest Newest Avg Sales
------ ------ ---------
1990 1992 52

EXAMPLE 2

Do the Example 1 query, but show the values only for autos that are Chevys:
SELECT 'Oldest'=MIN(year), 'Newest'=MAX(year),
'Avg Sales'=AVG(units_sold)
FROM automobile_sales_detail
WHERE model='Chevy'

Here's the result:

Oldest Newest Avg Sales
------ ------ ---------
1990 1992 56

EXAMPLE 3

Show the same values as in Example 1, but group them based on the model and color of the cars:
SELECT model, color, 'Oldest'=MIN(year), 'Newest'=MAX(year),
'Avg Sales'=AVG(units_sold)
FROM automobile_sales_detail
GROUP BY model, color
ORDER BY model, color

Here's the output:

model color Oldest Newest Avg Sales
----- ----- ------ ------ ---------
Chevy Blue 1990 1992 60
Chevy Red 1990 1992 30
Chevy White 1990 1992 78
Ford Blue 1990 1992 52
Ford Red 1990 1992 47
Ford White 1990 1992 44

EXAMPLE 4
This document is created with the unregistered version of CHM2PDF Pilot

Show the same values for only those model-year rows with average sales of 65 or fewer. Also, order and group the
output first by color and then by model, but keep the columns in the same order:
SELECT model, color, 'Oldest'=MIN(year), 'Newest'=MAX(year),
'Avg Sales'=AVG(units_sold)
FROM automobile_sales_detail
GROUP BY color, model HAVING AVG(units_sold) <= 65
ORDER BY color, model

Here's the output:

model color Oldest Newest Avg Sales
----- ----- ------ ------ ---------
Chevy Blue 1990 1992 60
Ford Blue 1990 1992 52
Chevy Red 1990 1992 30
Ford Red 1990 1992 47
Ford White 1990 1992 44

NOTE
I included an ORDER BY clause in these queries even though I specified that the criteria of the ORDER BY be the
same as for the GROUP BY clause. SQL Server has several alternative strategies for handling GROUP BY that don't
require sorting of the results, so the order in which the results are returned is unpredictable. If you want the results in a
particular order, you must specify ORDER BY.

An alternative to using COUNT(*)

You can use COUNT(*) to find the count of all the rows in the table, but a considerably faster way exists. The rows
column in the sysindexes table keeps the current rowcount dynamically for the clustered index. If one exists, indid =
1. If no clustered index exists, sysindexes keeps the count for the table (indid = 0). A faster query looks something
like this:
SELECT rows
FROM sysindexes
WHERE id=OBJECT_ID ('authors')
AND indid < 2

This query works only for base tables, not views, and it works only if you apply no selection criteria via a WHERE
clause. If you want to respond to a query such as "Show me the count of all rows for which the author lives in
California," you'd still need to use COUNT(*). In addition, there's no guarantee that this number will be accurate 100
percent of the time. In previous versions of SQL Server, certain operations (such as SELECT INTO), didn't always
correctly update the rows column in sysindexes. SQL Server 2000 is much more likely to keep sysindexes up to
date, but there's still no guarantee. In addition, you can execute the command DBCC UPDATEUSAGE to force the
values in sysindexes to be updated. However, there will be some nonzero lag time between running this command
and looking at the contents of sysindexes, and if the table is modified during that time, your result will still not be 100
percent guaranteed. If you absolutely must know exactly how many rows are in a table in your application, use the
COUNT(*) aggregate.

Aggregates and bigint

As the footnote to Table 7-7 indicates, numeric columns used in aggregate function can include the 8-byte integer
type bigint. However, you should keep in mind some special considerations when you work with bigint values. First
of all, Microsoft considers the 4-byte integer as the primary integer type; the 8-byte integer is something specialized.
Automatic promotions to bigint do not occur. When you use the aggregates AVG, SUM, MIN, and MAX on int,
This document is created with the unregistered version of CHM2PDF Pilot

smallint, or tinyint data, the result of the function is int. Only when the datatype of the expression operated upon by
these functions is bigint is the result of the function bigint. However, to avoid overflow of the internal counters,
internally SQL Server uses bigint values to process the AVG aggregate. Both the counter accumulating the total
number of rows and the counter accumulating the sum are stored internally as bigint.

With SUM, if there is a possibility that the result will be too large to fit into a 4-byte int, you should use the
CONVERT function to avoid an overflow error:
SELECT col1, SUM(CONVERT(bigint, col2))
FROM mytable
GROUP BY col1

COUNT always returns an int. If there is a possibility that the number of rows will exceed the maximum value of an
int (2^31-1), you can use the new aggregate function COUNT_BIG, which always returns a bigint.

Aggregate functions and NULLs

As usual, you must understand the effect of using NULLs to use them appropriately with aggregate functions.
Unfortunately, these effects aren't necessarily intuitive or even consistent. Because NULL is considered Unknown,
one could certainly argue that using SUM, MAX, MIN, or AVG on a table containing one or more NULL values
should also produce NULL. Obviously, if we have three employees and we don't know the salaries of two of them,
it's totally inaccurate to state that the SUM of the salaries for the three employees is equal to the one known salary.
Yet this is exactly how SUM, MIN, MAX, AVG, and COUNT work (but not COUNT(*)). Implicitly, these
functions seem to tack on the criterion of "for only those values that are not NULL."

You'll probably want these operations to work this way most of the time, and pragmatically, it's good that they do
because that means they're simpler to use in common situations. But if you don't understand exactly how these
functions will affect your code, you might introduce bugs. For example, if you divide SUM by the result of
COUNT(*) as a way to compute the mean value, the result probably won't be what you intended—SUM disregards
the NULL values, but COUNT(*) counts the rows that include those NULL values.

I've suggested a few times that using default or dummy values can be a good alternative to using NULL. But I've also
tried to make clear that while these alternatives solve some issues, they introduce other problems. You must account
for aggregates, and you might want to get back to exactly the semantics that aggregates such as SUM use with
defaults, ignoring the default or dummy values. Aggregating on only one column is pretty straightforward, and your
WHERE clause can simply exclude those values you want to disregard. But if you're aggregating on multiple columns,
eliminating rows in the WHERE clause might be a poor solution.

For example, suppose that in our employee table, we have both an emp_age column and an emp_salary column.
To avoid using NULL, we use 0.00 when we don't know the actual value. We want one query to do aggregations of
each column. Yet we don't want to disregard a row's value for emp_age if we don't have an actual value for
emp_salary, or vice-versa.

Here's our employee table:

emp_id emp_name emp_age emp_salary
------ -------- ------- ----------
1 Smith 34 26000.00
2 Doe 30 35000.00
3 Jones 45 0.00
4 Clark 0 65000.00
5 Kent 63 0.00

Now suppose we want to write a simple query to get the count of all employees, the average and lowest salaries, and
This document is created with the unregistered version of CHM2PDF Pilot

the average and lowest ages. Here's our first simplistic query:
SELECT
'Num Employees'=COUNT(*),
'Avg Salary'=AVG(emp_salary),
'Low Salary'=MIN(emp_salary),
'Avg Age'=AVG(emp_age),
'Youngest'=MIN(emp_age)
FROM employee

Here's the result:

Num Employees Avg Salary Low Salary Avg Age Youngest
------------- ---------- ---------- ------- --------
5 25200.00 0.00 34 0

Obviously, this query doesn't correctly answer our question because the dummy values of 0 have distorted the results
for the AVG and MIN columns. So we decide to exclude those rows by stating in the WHERE clause that the salary
should not be the default value:
SELECT
'Num Employees'=COUNT(*),
'Avg Salary'=AVG(emp_salary),
'Low Salary'=MIN(emp_salary),
'Avg Age'=AVG(emp_age),
'Youngest'=MIN(emp_age)
FROM employee
WHERE emp_salary > 0 AND emp_age > 0

Here's what we get:

Num Employees Avg Salary Low Salary Avg Age Youngest
------------- ---------- ---------- ------- --------
2 30500.00 26000.00 32 30

This query is marginally better because at least the average salary is no longer lower than any actually known salary.
The values are based on only the first two employees, Smith and Doe—we have five employees, not just two, as this
result claims. Two other employees, Jones and Kent, are significantly older than the two who were used to compute
the average age. And had Clark's salary been considered, the average salary would be much higher. We are
seemingly stuck. It looks as if we can't write a single query to answer the question and might have to write separate
queries to get the count, the salary information, and the age information.

Fortunately, the NULLIF function comes to the rescue. This function is one of many handy Transact-SQL functions.
NULLIF takes two expressions, NULLIF(expression1, expression2), and it returns NULL if the two expressions
are equivalent and expression1 if they aren't equivalent. The NULLIF function is roughly the mirror image of another
function, ISNULL, which produces a value if a NULL is encountered (as discussed earlier in this chapter). The
NULLIF function is actually a special-purpose shorthand of the ANSI SQL-92 CASE expression (discussed in detail
in Chapter 10). Using NULLIF is equivalent to the following:
CASE
WHEN expression1=expression2 THEN NULL
ELSE expression1
END

Hence, NULLIF(emp_salary, 0) produces NULL for rows in which the salary equals 0. By converting the 0 values
This document is created with the unregistered version of CHM2PDF Pilot

to NULL, we can use the aggregate functions just as we'd use them with NULL so that SUM and MIN disregard the
NULL entries.
SELECT
'Num Employees'=COUNT(*),
'Avg Salary'=AVG(NULLIF(emp_salary, 0)),
'Low Salary'=MIN(NULLIF(emp_salary, 0)),
'Avg Age'=AVG(NULLIF(emp_age, 0)),
'Youngest'=MIN(NULLIF(emp_age, 0))
FROM employee

Here's the more accurate result:

Num Employees Avg Salary Low Salary Avg Age Youngest
------------- ---------- ---------- ------- --------
5 42000.00 26000.00 43 30

Finally! This result is exactly what we want, and all five employees are represented. Of the three whose salaries we
know, the average salary is $42,000.00 and the low salary is $26,000.00. Of the four whose ages we know, the
average age is 43 and the youngest is 30.

Datacube—Aggregate Variations
The previous aggregate examples in this chapter, which all use standard SQL-92 syntax, show that by formulating a
specific query on the sample table with one of the aggregate functions, we can answer almost any question regarding
aggregate sales for some combination of model, year, and color of car. However, answering any of these questions
requires a separate, specific query. Of course, it's common to look at data and keep formulating and posing new and
different questions. Such questions are the hallmark of data mining, decision-support systems (DSSs), online analytic
processing (OLAP), or whatever other name is used these days for this age-old need to examine data.

To address this inherent weakness in standard SQL, developers have created several front-end tools that maintain a
set of data that allow you to easily query changes in aggregation, thereby eliminating the need to sweep through data
for every new aggregation request. One such tool is Microsoft Analysis Services, which you can install from the same
CD that contains your SQL Server 2000 software. Analysis Services tools can add a lot of value, and they make it
easy to slice and dice the values pertaining to any "cut" of the data groupings. But even these tools can perform better
with help from the database engine.

SQL Server has two extensions to GROUP BY—CUBE and ROLLUP—that allow SQL Server to optionally
generate all the aggregate groupings in a single query.

CUBE
CUBE explodes data to produce a result set containing a superset of groups, with a cross-tabulation of every column
to the value of every other column, as well as a special super-aggregate value that can be thought of as meaning ALL
VALUES. Here's the CUBE for the 18 rows of the automobile_sales_detail table. Notice that these 18 rows
generate 48 rows in the datacube.
units model year color
----- ----- ---- -----
62 Chevy 1990 Blue
5 Chevy 1990 Red
87 Chevy 1990 White
154 Chevy 1990 ALL
49 Chevy 1991 Blue
54 Chevy 1991 Red
This document is created with the unregistered version of CHM2PDF Pilot

95 Chevy 1991 White

198 Chevy 1991 ALL
71 Chevy 1992 Blue
31 Chevy 1992 Red
54 Chevy 1992 White
156 Chevy 1992 ALL
508 Chevy ALL ALL
63 Ford 1990 Blue
64 Ford 1990 Red
62 Ford 1990 White
189 Ford 1990 ALL
55 Ford 1991 Blue
52 Ford 1991 Red
9 Ford 1991 White
116 Ford 1991 ALL
39 Ford 1992 Blue
27 Ford 1992 Red
62 Ford 1992 White
128 Ford 1992 ALL
433 Ford ALL ALL
941 ALL ALL ALL
125 ALL 1990 Blue
69 ALL 1990 Red
149 ALL 1990 White
343 ALL 1990 ALL
104 ALL 1991 Blue
106 ALL 1991 Red
104 ALL 1991 White
314 ALL 1991 ALL
110 ALL 1992 Blue
58 ALL 1992 Red
116 ALL 1992 White
284 ALL 1992 ALL
182 Chevy ALL Blue
157 Ford ALL Blue
339 ALL ALL Blue
90 Chevy ALL Red
143 Ford ALL Red
233 ALL ALL Red
236 Chevy ALL White
133 Ford ALL White
369 ALL ALL White

The results of the CUBE make it simple to answer just about any sales aggregation question that you can think of. If
you want to get sales information for all 1992 automobiles, regardless of model or color, you can find the
corresponding row and see that 284 cars were sold. To find all Chevy sales for all years and colors, it's equally easy
to find 508. Such a result set might be used as is for a reference chart or in coordination with a Data Mining tool.
Some History
The motivation for inventing CUBE came from a paper written by Jim Gray and others soon after Jim joined
Microsoft as a researcher. After reading his paper, a few of the developers got excited about adding the feature to
SQL Server 6.5, and Don Reichart did a masterful job of implementing this important feature in record time.
Subsequently, this feature was submitted to the ANSI SQL committee as a proposed addition to the standard. For
more details about CUBE, you can read the paper that Jim Gray submitted to the ACM on the companion CD.

The number of rows exploded by a CUBE operation can be surprisingly large. In the example above, 18 rows
generated 48 rows of output. However, the result of a CUBE operation won't necessarily produce more rows of
output than what appears in the underlying data. The number of rows generated will depend on the number of
attributes being grouped and the actual combinations in your data. Without detailed knowledge of the data, you can't
predict the number of rows produced by a CUBE operation. However, the upper bound can be easily predicted. The
upper bound will be equal to the cross-product of the (number of distinct values + 1) value for each attribute. The
This document is created with the unregistered version of CHM2PDF Pilot

addition of 1 is for the case of "ALL." The automobile_sales_detail example has three attributes: model, year, and
color.
Upper bound of number of rows
= (Number of models + 1) * (Number of years + 1) *
(Number of colors + 1)
= (2 + 1) * (3 + 1) * (3 + 1)
= 48

This example should make clear that the upper bound of the number of rows depends not on the number of data
rows, but rather on the number of attributes being grouped and the number of distinct values for each attribute. For
example, if the table had 18 million instead of 18 data rows but still had no more than two models, three years, and
three colors (there's no stated UNIQUE or PRIMARY KEY constraint on the model, year, or color fields), the
CUBE operation will again return at most only 48 rows.

The actual number of rows might be considerably less than the upper bound, however, because CUBE doesn't try to
force a 0 or a NULL aggregate for combinations for which no values exist. In the carefully constructed example you
just saw, there's complete coverage across every attribute. Each of the two models has an entry for each of the three
years and for each of the three colors. But suppose that no sales data appears for the 1990 Chevy and that no red
Fords exist for any year. The raw data would look like this:
Rowid Model Year Color Units_Sold
----- ----- ---- ----- ----------
1 Chevy 1991 Red 54
2 Chevy 1991 White 95
3 Chevy 1991 Blue 49
4 Chevy 1992 Red 31
5 Chevy 1992 White 54
6 Chevy 1992 Blue 71
7 Ford 1990 White 62
8 Ford 1990 Blue 63
9 Ford 1991 White 9
10 Ford 1991 Blue 55
11 Ford 1992 White 62
12 Ford 1992 Blue 39

Here's the new cube:

Units Model Year Color
----- ----- ---- -----
49 Chevy 1991 Blue
54 Chevy 1991 Red
95 Chevy 1991 White
198 Chevy 1991 ALL
71 Chevy 1992 Blue
31 Chevy 1992 Red
54 Chevy 1992 White
156 Chevy 1992 ALL
354 Chevy ALL ALL
63 Ford 1990 Blue
62 Ford 1990 White
125 Ford 1990 ALL
55 Ford 1991 Blue
9 Ford 1991 White
64 Ford 1991 ALL
39 Ford 1992 Blue
62 Ford 1992 White
101 Ford 1992 ALL
290 Ford ALL ALL
644 ALL ALL ALL
This document is created with the unregistered version of CHM2PDF Pilot

63 ALL 1990 Blue

62 ALL 1990 White
125 ALL 1990 ALL
104 ALL 1991 Blue
54 ALL 1991 Red
104 ALL 1991 White
262 ALL 1991 ALL
110 ALL 1992 Blue
31 ALL 1992 Red
116 ALL 1992 White
257 ALL 1992 ALL
120 Chevy ALL Blue
157 Ford ALL Blue
277 ALL ALL Blue
85 Chevy ALL Red
85 ALL ALL Red
149 Chevy ALL White
133 Ford ALL White
282 ALL ALL White

We now have 12 data rows but still three attributes and the same number of distinct values for each attribute. But
instead of 48 rows for the result of the CUBE, we have only 39 rows because some combinations of attributes have
no data. For example, no row exists for all models of red cars for 1990 because no data met that specific criterion.

If you would benefit from always having the full cube populated when you use CUBE, you might want to add a
placeholder row with dummy values for the columns being aggregated so that every combination is represented. Or
you can run a query that produces the cross-product of all combinations that do not exist, and then UNION the result
with the cube. (I'll show you an example of this later when I discuss UNION.) Be careful: as the number of attributes
and the cardinality between the values increases, the cube can quickly get large. The full cube based on five attributes,
each having 50 distinct values, with at least one matching entry for each combination, generates (50 + 1)5, or
345,025,251, rows in the cube. If coverage of all these combinations is nowhere near full, the cube might generate
"only" a few thousand rows instead of more than 345 million!

We haven't seen the actual query used to produce the cube because you need to understand the basic concepts
before yet again dealing with issues of NULL. A simple GROUP BY column WITH CUBE does not, by default,
produce super-aggregate rows with a value that outputs as "ALL." The ALL value is really a nonvalue, like NULL
(gasp!). In fact, the ALL column is represented by (gulp!) a special kind of NULL, referred to as a grouping NULL.

While implementing this new feature, the developers debated whether to use NULL or invent a new marker value
similar in some ways to NULL but clearly still separate from data. The decision wasn't easy. A new data value would
be more theoretically pure. The meaning of "ALL" is really different from "unknown" or "missing"—that is, it's different
from NULL. However, had they introduced another special-meaning marker, it would have required the same level of
complexity as NULL—when working with special operators like IS ALL, for example—and the truth tables would
be all the more difficult to work with in expressions such as =, <, IN, and so on. And the developers did have some
precedents that they wanted to avoid.

In many ways, the longtime SQL Server feature COMPUTE BY is quite similar to CUBE. (Actually, it's more similar
to a derivative of CUBE called ROLLUP, which we'll see shortly.) COMPUTE BY, although helpful, is rarely used,
largely because it doesn't generate a standard row format for the result set but rather returns a special alternate
result set. Applications must go to significant lengths to handle this special-purpose result, which is nonrelational in
that the output itself isn't a table. Most applications haven't been able or willing to go the distance and have therefore
not supported COMPUTE BY. Because of this, the SQL Server development team chose to overload NULL, which
applications already have to deal with to some extent. (This presents a few wrinkles of its own, which I'll come back
to.)

The following query performs a basic CUBE operation on the automobile_sales_detail table:
This document is created with the unregistered version of CHM2PDF Pilot

SELECT SUM(units_sold), model, year, color

FROM automobile_sales_detail
GROUP BY model, year, color WITH CUBE

Here's the output:

model year color
----------- ----------- ----------- -----------
62 Chevy 1990 Blue
5 Chevy 1990 Red
87 Chevy 1990 White
154 Chevy 1990 NULL
49 Chevy 1991 Blue
54 Chevy 1991 Red
95 Chevy 1991 White
198 Chevy 1991 NULL
71 Chevy 1992 Blue
31 Chevy 1992 Red
54 Chevy 1992 White
156 Chevy 1992 NULL
508 Chevy NULL NULL
63 Ford 1990 Blue
64 Ford 1990 Red
62 Ford 1990 White
189 Ford 1990 NULL
55 Ford 1991 Blue
52 Ford 1991 Red
9 Ford 1991 White
116 Ford 1991 NULL
39 Ford 1992 Blue
27 Ford 1992 Red
62 Ford 1992 White
128 Ford 1992 NULL
433 Ford NULL NULL
941 NULL NULL NULL
125 NULL 1990 Blue
69 NULL 1990 Red
149 NULL 1990 White
343 NULL 1990 NULL
104 NULL 1991 Blue
106 NULL 1991 Red
104 NULL 1991 White
314 NULL 1991 NULL
110 NULL 1992 Blue
58 NULL 1992 Red
116 NULL 1992 White
284 NULL 1992 NULL
182 Chevy NULL Blue
157 Ford NULL Blue
339 NULL NULL Blue
90 Chevy NULL Red
143 Ford NULL Red
233 NULL NULL Red
236 Chevy NULL White
133 Ford NULL White
369 NULL NULL White

Notice that rather than ALL, which we had constructed earlier to be a placeholder, the super-aggregate values are
represented by NULL—here formatted as NULL as in SQL Query Analyzer, although the actual display of a NULL
is application specific. The data in this case has no NULL values, so we can easily tell the difference between NULL
meaning "unknown" and NULL meaning "ALL."
This document is created with the unregistered version of CHM2PDF Pilot

If your data uses some NULL values, however, you won't be so lucky. Suppose we add one row of data that has
NULL for both model and year (assuming that these columns allow NULL):
INSERT automobile_sales_detail values (NULL, NULL, 'White', 10)

The last row in the CUBE above (which has 369 as total sales for all white cars) would be seemingly indistinguishable
from this new data row. Fortunately, the GROUPING function comes to the rescue to differentiate between the two.
It returns 1 (TRUE) if the element is an ALL value and 0 (FALSE) otherwise. Here is a modified query to show the
CUBE with the inclusion of the row that has NULL values and with the addition of the GROUPING function to
designate ALL values:
SELECT 'Units Sold'=SUM(units_sold),
model, 'ALL Models'=GROUPING(model),
year, 'ALL Years'=GROUPING(year),
color, 'ALL Colors'=GROUPING(color)
FROM automobile_sales_detail
GROUP BY model, year, color WITH CUBE

Here's the result set:

ALL ALL ALL
Units Sold model Models year Years color Colors
---------- ----- ------ ------ ----- ----- ------
10 NULL 0 NULL 0 White 0
10 NULL 0 NULL 0 NULL 1
10 NULL 0 NULL 1 NULL 1
62 Chevy 0 1990 0 Blue 0
5 Chevy 0 1990 0 Red 0
87 Chevy 0 1990 0 White 0
154 Chevy 0 1990 0 NULL 1
49 Chevy 0 1991 0 Blue 0
54 Chevy 0 1991 0 Red 0
95 Chevy 0 1991 0 White 0
198 Chevy 0 1991 0 NULL 1
71 Chevy 0 1992 0 Blue 0
31 Chevy 0 1992 0 Red 0
54 Chevy 0 1992 0 White 0
156 Chevy 0 1992 0 NULL 1
508 Chevy 0 NULL 1 NULL 1
63 Ford 0 1990 0 Blue 0
64 Ford 0 1990 0 Red 0
62 Ford 0 1990 0 White 0
189 Ford 0 1990 0 NULL 1
55 Ford 0 1991 0 Blue 0
52 Ford 0 1991 0 Red 0
9 Ford 0 1991 0 White 0
116 Ford 0 1991 0 NULL 1
39 Ford 0 1992 0 Blue 0
27 Ford 0 1992 0 Red 0
62 Ford 0 1992 0 White 0
128 Ford 0 1992 0 NULL 1
433 Ford 0 NULL 1 NULL 1
951 NULL 1 NULL 1 NULL 1
10 NULL 1 NULL 0 White 0
10 NULL 1 NULL 0 NULL 1
125 NULL 1 1990 0 Blue 0
69 NULL 1 1990 0 Red 0
149 NULL 1 1990 0 White 0
343 NULL 1 1990 0 NULL 1
104 NULL 1 1991 0 Blue 0
This document is created with the unregistered version of CHM2PDF Pilot

106 NULL 1 1991 0 Red 0

104 NULL 1 1991 0 White 0
314 NULL 1 1991 0 NULL 1
110 NULL 1 1992 0 Blue 0
58 NULL 1 1992 0 Red 0
116 NULL 1 1992 0 White 0
284 NULL 1 1992 0 NULL 1
182 Chevy 0 NULL 1 Blue 0
157 Ford 0 NULL 1 Blue 0
339 NULL 1 NULL 1 Blue 0
90 Chevy 0 NULL 1 Red 0
143 Ford 0 NULL 1 Red 0
233 NULL 1 NULL 1 Red 0
10 NULL 0 NULL 1 White 0
236 Chevy 0 NULL 1 White 0
133 Ford 0 NULL 1 White 0
379 NULL 1 NULL 1 White 0

Note that the GROUPING function takes a column name as an argument. A 0 returned for GROUPING(
column_name) means that the value returned for that column is a real value. A 1 returned for GROUPING(
column_name) means that the value returned for that column is not real—it was returned only because of the use of
the CUBE or ROLLUP option.

The GROUPING function enables you to differentiate between a GROUPING NULL value and a NULL value, but
it hardly makes reading and analyzing the results returned by CUBE and ROLLUP as intuitive as when you use ALL.
An alternative is to write a view that outputs a grouping value as ALL. You might be more comfortable turning your
programmers and power users loose on such a view rather than relying on them understanding GROUPING NULL
vs. NULL. This example uses the CASE expression, which is discussed in detail in Chapter 10:
CREATE VIEW auto_cube (units, model, year, color) AS
SELECT SUM(units_sold),
CASE WHEN (GROUPING(model)=1) THEN 'ALL'
ELSE ISNULL(model, '????')
END,
CASE WHEN (GROUPING(year)=1) THEN 'ALL'
ELSE ISNULL(CONVERT(char(6), year), '????')
END,
CASE WHEN (GROUPING(color)=1) THEN 'ALL'
ELSE ISNULL(color, '????')
END
FROM automobile_sales_detail
GROUP BY model, year, color WITH CUBE

With this view, it's simple for someone to understand the difference between grouping values (represented by "ALL")
and NULL data values (represented by "????"), and to easily refine the query if necessary to look for certain data.
Here's a simple query:
SELECT * FROM auto_cube

Here's the result:

units model year color
----- ----- ---- -----
10 ???? ???? White
10 ???? ???? ALL
10 ???? ALL ALL
62 Chevy 1990 Blue
5 Chevy 1990 Red
87 Chevy 1990 White
This document is created with the unregistered version of CHM2PDF Pilot

154 Chevy 1990 ALL

49 Chevy 1991 Blue
54 Chevy 1991 Red
95 Chevy 1991 White
198 Chevy 1991 ALL
71 Chevy 1992 Blue
31 Chevy 1992 Red
54 Chevy 1992 White
156 Chevy 1992 ALL
508 Chevy ALL ALL
63 Ford 1990 Blue
64 Ford 1990 Red
62 Ford 1990 White
189 Ford 1990 ALL
55 Ford 1991 Blue
52 Ford 1991 Red
9 Ford 1991 White
116 Ford 1991 ALL
39 Ford 1992 Blue
27 Ford 1992 Red
62 Ford 1992 White
128 Ford 1992 ALL
433 Ford ALL ALL
951 ALL ALL ALL
10 ALL ???? White
10 ALL ???? ALL
125 ALL 1990 Blue
69 ALL 1990 Red
149 ALL 1990 White
343 ALL 1990 ALL
104 ALL 1991 Blue
106 ALL 1991 Red
104 ALL 1991 White
314 ALL 1991 ALL
110 ALL 1992 Blue
58 ALL 1992 Red
116 ALL 1992 White
284 ALL 1992 ALL
182 Chevy ALL Blue
157 Ford ALL Blue
339 ALL ALL Blue
90 Chevy ALL Red
143 Ford ALL Red
233 ALL ALL Red
10 ???? ALL White
236 Chevy ALL White
133 Ford ALL White
379 ALL ALL White

Working with this view is easy. Grouping values appear as "ALL." Actual NULL values are represented by question
marks, which is appropriate because they're unknown. You can easily further select from the view to drill into any
dimension of the cube. For example, this query finds all Chevy sales, regardless of the year and color:
SELECT * FROM auto_cube
WHERE model='Chevy' AND year='ALL' AND color='ALL'

Here's the result:

units model year color
----- ----- ---- -----
508 Chevy ALL ALL
This document is created with the unregistered version of CHM2PDF Pilot

In this view, I elected to output everything as character data; thus, "ALL" was a reasonable choice. If keeping the
data numeric had been important, I could've chosen some special value, such as -999999.

ROLLUP
If you're looking for a hierarchy or a drill-down report (what you might know as a control-break report if you've
ever programmed in COBOL), CUBE can be overkill. It generates many more result rows than you want, and it
obscures the requested information with more data. (Don't be confused into thinking that data and information are
equivalent.) When you use CUBE, you get all permutations, including super-aggregates, for all attributes for which
corresponding data exists. SQL Server provides the ROLLUP operator to extract statistics and summary information
from result sets. ROLLUP returns only the values for a hierarchy of the attributes that you specify.

This behavior is best explained with an example. If we change the CUBE query to use ROLLUP instead, the results
are more compact and easier to interpret when we drill into progressive levels of detail for sales by model. However,
with ROLLUP, we don't get one-stop shopping to answer a question like "How many white cars of any model were
sold?"
SELECT 'units sold'=SUM(units_sold),
'model'=CASE WHEN (GROUPING(model)=1) THEN 'ALL'
ELSE ISNULL(model, '????')
END,
'year'=CASE WHEN (GROUPING(year)=1) THEN 'ALL'
ELSE ISNULL(CONVERT(char(6), year), '????')
END,
'color'=CASE WHEN (GROUPING(color)=1) THEN 'ALL'
ELSE ISNULL(color, '????')
END
FROM automobile_sales_detail
GROUP BY model, year, color WITH ROLLUP

The results:
units sold model year color
---------- ----- ---- -----
10 ???? ???? White
10 ???? ???? ALL
10 ???? ALL ALL
62 Chevy 1990 Blue
5 Chevy 1990 Red
87 Chevy 1990 White
154 Chevy 1990 ALL
49 Chevy 1991 Blue
54 Chevy 1991 Red
95 Chevy 1991 White
198 Chevy 1991 ALL
71 Chevy 1992 Blue
31 Chevy 1992 Red
54 Chevy 1992 White
156 Chevy 1992 ALL
508 Chevy ALL ALL
63 Ford 1990 Blue
64 Ford 1990 Red
62 Ford 1990 White
189 Ford 1990 ALL
55 Ford 1991 Blue
52 Ford 1991 Red
9 Ford 1991 White
116 Ford 1991 ALL
39 Ford 1992 Blue
This document is created with the unregistered version of CHM2PDF Pilot

27 Ford 1992 Red

62 Ford 1992 White
128 Ford 1992 ALL
433 Ford ALL ALL
951 ALL ALL ALL

ROLLUP is similar to COMPUTE BY, which has nice functionality. However, COMPUTE BY is not as relational
as ROLLUP—it doesn't simply produce results as an ordinary table of values. COMPUTE BY always requires
special application programming to deal with its alternate result sets, which are essentially equivalent to
super-aggregates. Recall that CUBE and ROLLUP simply use ordinary result sets and that the super-aggregates are
represented as a grouping NULL. Generally speaking, CUBE and ROLLUP are preferable to COMPUTE BY
because they're easier to fit into applications; you can use them with views, CASE, subselects, and column aliases;
and they're internally better optimized. But for completeness, here's an example that uses COMPUTE BY as
functionally equivalent to ROLLUP:
SELECT units_sold, model, year, color
FROM automobile_sales_detail
ORDER BY model, year, color
COMPUTE SUM(units_sold) BY model, year
COMPUTE SUM(units_sold) BY model
COMPUTE SUM(units_sold)

Here's the result:

units_sold model year color
---------- ----- ---- -----
10 NULL NULL White

Sum
===========
10
Sum
===========
10

units_sold model year color

---------- ----- ---- -----
62 Chevy 1990 Blue
5 Chevy 1990 Red
87 Chevy 1990 White

Sum
===========
154

units_sold model year color

---------- ----- ---- -----
49 Chevy 1991 Blue
54 Chevy 1991 Red
95 Chevy 1991 White

Sum
===========
198

units_sold model year color

---------- ----- ---- -----
71 Chevy 1992 Blue
31 Chevy 1992 Red
54 Chevy 1992 White
This document is created with the unregistered version of CHM2PDF Pilot

Sum
===========
156
Sum
===========
508

units_sold model year color

---------- ----- ---- -----
63 Ford 1990 Blue
64 Ford 1990 Red
62 Ford 1990 White

Sum
===========
189

units_sold model year color

---------- ----- ---- -----
55 Ford 1991 Blue
52 Ford 1991 Red
9 Ford 1991 White

Sum
===========
116

units_sold model year color

---------- ----- ---- -----
39 Ford 1992 Blue
27 Ford 1992 Red
62 Ford 1992 White

Sum
===========
128
Sum
===========
433
Sum
===========
951

Statistical aggregates
SQL Server 2000 includes four statistical aggregate functions in addition to the standard six, plus COUNT_BIG,
described earlier. These functions are for statistical applications and are considered aggregate functions because they
operate on a set of values instead of on a single value. Table 7-8 describes the four statistical aggregate functions.

You can find the standard deviation and variance of the prices in the titles table with this query, for example:
SELECT STDEV(price), VAR(price)
FROM titles

Alternatively, you can calculate the standard deviation and variance of the prices for each different type of book:
SELECT type, STDEV(price), VAR(price)
FROM titles
GROUP BY type
This document is created with the unregistered version of CHM2PDF Pilot

Table 7-8. Statistical aggregate functions.

Aggregate Function Description
Returns the statistical standard deviation of all values in
STDEV(expression)
the given expression.
Returns the statistical standard deviation for the
STDEVP(expression)
population for all values in the given expression.
Returns the statistical variance of all values in the given
VAR(expression)
expression.
Returns the statistical variance for the population for all
VARP(expression)
values in the given expression.

TOP
You've already seen that the WHERE clause allows you to limit the number of rows that a SELECT statement will
return. But the WHERE clause assumes that you have knowledge of the actual data values present. What if you want
to see only a screenful of rows from a table but you have no idea which range of values are present in the table? Prior
to SQL Server version 7, you could use the option SET ROWCOUNT n to limit SQL Server to sending only n rows
to the client. Every SELECT statement would just stop sending rows back to the client after n rows had been sent.

For example, the following query returns only 10 rows from the authors table:
SET ROWCOUNT 10
GO
SELECT * FROM authors
GO

This is useful in some situations, but it's quite limited in functionality. Using the TOP keyword in the SELECT list
allows much greater control over the quantity of data you want to see. Here's the syntax for using TOP:

SELECT [ TOP n [PERCENT] [ WITH TIES] ] select_list

FROM table_list
WHERE conditions
...rest of query

When the query includes an ORDER BY clause, TOP causes the first n rows ordered by the ORDER BY clause to
be output. When the query has no ORDER BY clause, you can't predict which rows will be returned. Compare the
output of the following two queries:

QUERY 1
SELECT TOP 5 title_id, price, type
FROM titles

title_id price type

-------- --------------------- ------------
BU1032 19.9900 business
BU1111 11.9500 business
BU2075 2.9900 business
BU7832 19.9900 business
MC2222 19.9900 mod_cook

QUERY 2
This document is created with the unregistered version of CHM2PDF Pilot

SELECT TOP 5 title_id, price, type

FROM titles
ORDER BY price DESC

title_id price type

-------- --------------------- ------------
PC1035 22.9500 popular_comp
PS1372 21.5900 psychology
TC3218 20.9500 trad_cook
PC8888 20.0000 popular_comp
BU1032 19.9900 business

Notice that simply including TOP 5 in the SELECT list is functionally equivalent to setting ROWCOUNT to 5. SQL
Server just stops returning rows after the first five. I say "functionally equivalent" because you get the same rows back
using either method. However, the SQL Server query processor can optimize queries using TOP, and in many cases
chooses a more efficient query plan if it knows during optimization that only a few rows are needed.

In the example above, other books exist with a price of $19.99, but because we wanted only five, we didn't see
them. Does this output really answer the question "What are the five highest priced books?" If not, and you really
want to see all the books with the same price as the ones listed, you can use the WITH TIES option. This option is
allowed only if your SELECT statement includes an ORDER BY:
SELECT TOP 5 WITH TIES title_id, price, type
FROM titles
ORDER BY price DESC

The query returns the following eight rows:

title_id price type
-------- --------------------- ------------
PC1035 22.9500 popular_comp
PS1372 21.5900 psychology
TC3218 20.9500 trad_cook
PC8888 20.0000 popular_comp
BU1032 19.9900 business
BU7832 19.9900 business
MC2222 19.9900 mod_cook
PS3333 19.9900 psychology

You can think of SQL Server carrying out this operation by first counting out the number of rows specified by
TOP—in this case, five. Only after the first five are returned does SQL Server check to see whether any other rows
have a value equivalent to the last one returned.

This doesn't necessarily return the books with the five highest prices. If we change TOP 5 to TOP 8, we get the same
eight rows back, which represent only five different prices. I compare this to a race in which ribbons are awarded to
the first five finishers. If four participants tie for fifth place, coming in at exactly the same time, we have to award more
than five ribbons. Note that only ties at the end cause extra rows to come back. In the race analogy, if we have four
racers tied for first place and the fifth-fastest racer comes in alone, we can still award just five ribbons. There are only
two different times for the five winners.

If you really want to see the eight highest prices (or eight fastest times), you can use DISTINCT with TOP. You can't
select title_id at the same time because DISTINCT applies to the whole result row and every title_id and price
combination is distinct in itself. However, you can use a subquery to determine which rows have one of the eight
highest prices:
SELECT title_id, price, type
This document is created with the unregistered version of CHM2PDF Pilot

FROM titles
WHERE price in (SELECT DISTINCT TOP 8 price
FROM titles
ORDER BY price DESC)

In this case, the rows won't come back in descending order of price because the order is controlled by the outer
query. To get this, you have to add another ORDER BY at the end.

If you want to see a certain fraction of the rows, you can use TOP with PERCENT, which rounds up to the nearest
integer number of rows:
SELECT TOP 30 Percent title_id, price, type
FROM titles
ORDER BY price DESC

The titles table contains 18 rows, and 30 percent of 18 is 5.6. This is rounded up to return these six rows:
title_id price type
-------- --------------------- ------------
PC1035 22.9500 popular_comp
PS1372 21.5900 psychology
TC3218 20.9500 trad_cook
PC8888 20.0000 popular_comp
BU1032 19.9900 business
BU7832 19.9900 business

You can also use the WITH TIES option with PERCENT. If you use it in the above query, you get the additional
two rows that also have a price of $19.99.

UNION
UNION is conceptually easy to understand. Given the same number of columns and compatible datatypes, UNION
combines two or more result sets into a single result set. By default, duplicate rows are eliminated, although you can
include them by specifying UNION ALL. The datatypes of the result sets to be combined with UNION don't have to
be identical, but they must be able to be implicitly converted. If this isn't the case, you can explicitly convert them to
identical or compatible types using the handy CONVERT function.

The following example shows UNION at its most basic level. It also demonstrates that a SELECT statement in SQL
Server doesn't need to be from a table or a view but can be a constant or a variable:
SELECT col1=2, col2=1
UNION
SELECT xxx=1, yyy=2
UNION
SELECT '3', '0'
UNION
SELECT TAN(0), 3*1000/10/100 -- Tangent of 0 is 0.
-- So this row produces 0.0, 3
UNION
SELECT 1, 2

The output follows:

col1 col2
---- ----
This document is created with the unregistered version of CHM2PDF Pilot

0.0 3
3.0 0
1.0 2
2.0 1

Versions of SQL Server prior to SQL Server 7 always returned the result of a UNION in sorted order unless you
specified UNION ALL. This occurred because UNION eliminated duplicate rows using a sorting strategy. The
ordering was simply a by-product of the sorting to eliminate duplicates. However, SQL Server 2000 has several
alternative strategies available for removing duplicates, so there's no guarantee of any particular order when you use
UNION. If order is important to you, you should explicitly use ORDER BY at the end of the last SELECT statement.

In the example above, the datatypes are compatible but not identical. Because TAN(0) returns a float datatype, col1
is automatically cast to float (since using an integer would lose precision), and the other values are implicitly converted.
In addition, the one duplicate row (1.0, 2) is eliminated, and the columns take their monikers from the first result set.

If you use UNION ALL instead of UNION, no extra work is needed to find and eliminate any duplicate rows. If
there's no need to eliminate duplicates, or if you know that there won't be any duplicates, you can achieve a
noticeable performance improvement by using UNION ALL. The following query illustrates the use of UNION ALL
and uses ORDER BY to present the results in a particular order. It also converts the result of the tangent operation to
a tinyint so that col1 is then implicitly converted to an int.
SELECT col1=2, col2=1
UNION ALL
SELECT xxx=1, yyy=2
UNION ALL
SELECT '3', '0'
UNION ALL
SELECT CONVERT(smallint, TAN(0)), 3*1000/10/100
-- Tangent of 0 is 0. So this row produces 0.0, 3
UNION ALL
SELECT 1, 2
ORDER BY col2

Here's the output:

col1 col2
---- ----
3 0
2 1
1 2
1 2
0 3

Earlier you saw that the CUBE operation doesn't generate missing rows with the value 0. However, you might
occasionally want the full cube, with 0 as the SUM value for missing combinations. I promised you an example of how
this can be accomplished using UNION, so here's the example. And here's the key: the table can be cross-joined to
itself to produce a 0 row for every combination possible given the non-NULL data. This query uses the existing data
to produce a list of every possible combination of model, color, and year:
SELECT DISTINCT units_sold=0, A.model, B.color, C.year
FROM
automobile_sales_detail A
CROSS JOIN
automobile_sales_detail B
CROSS JOIN
automobile_sales_detail C
This document is created with the unregistered version of CHM2PDF Pilot

Here's the output:

units_sold model color year
---------- ----- ----- ----
0 Chevy Blue 1990
0 Chevy Blue 1991
0 Chevy Blue 1992
0 Chevy Red 1990
0 Chevy Red 1991
0 Chevy Red 1992
0 Chevy White 1990
0 Chevy White 1991
0 Chevy White 1992
0 Ford Blue 1990
0 Ford Blue 1991
0 Ford Blue 1992
0 Ford Red 1990
0 Ford Red 1991
0 Ford Red 1992
0 Ford White 1990
0 Ford White 1991
0 Ford White 1992

The real underlying data is the set of 12 rows shown earlier and shown again here. Recall that six combinations have
no value—that is, there are no Chevys for 1990 and no red Fords:
model year color units_sold
----- ---- ----- ----------
Chevy 1991 Red 54
Chevy 1991 White 95
Chevy 1991 Blue 49
Chevy 1992 Red 31
Chevy 1992 White 54
Chevy 1992 Blue 71
Ford 1990 White 62
Ford 1990 Blue 63
Ford 1991 White 9
Ford 1991 Blue 55
Ford 1992 White 62
Ford 1992 Blue 39

Having generated all the dummy rows as such, we can then easily UNION the cross-joined dummy results with the
output of the CUBE. But that still won't do because we get both a real row and a dummy row whenever a real
combination exists. SQL Server provides several good ways to solve this dilemma. We can use NOT EXISTS to
produce the dummy rows only for combinations that don't exist. Another approach, and perhaps the most intuitive
one, is to make a view of the cross-joined dummy rows and the actual data, and then perform the CUBE on the view.
The 0 values, of course, don't affect the SUM, so this works nicely:
CREATE VIEW fullcube
(
units_sold,
model,
year,
color
)
AS
(
SELECT D.units_sold, D.model, D.year, D.color
FROM automobile_sales_detail D
UNION ALL
This document is created with the unregistered version of CHM2PDF Pilot

SELECT DISTINCT 0,
A.model, C.year, B.color
FROM
automobile_sales_detail A
CROSS JOIN
automobile_sales_detail B
CROSS JOIN
automobile_sales_detail C
)

Having constructed the view, we can then issue our CUBE query against it, just as we did previously against the base
table:
SELECT units_sold=SUM(units_sold), model, year, color
FROM fullcube
GROUP BY model, year, color WITH CUBE

Here's the output:

units_sold model year color
---------- ----- ---- -----
0 Chevy 1990 Blue
0 Chevy 1990 Red
0 Chevy 1990 White
0 Chevy 1990 NULL
49 Chevy 1991 Blue
54 Chevy 1991 Red
95 Chevy 1991 White
198 Chevy 1991 NULL
71 Chevy 1992 Blue
31 Chevy 1992 Red
54 Chevy 1992 White
156 Chevy 1992 NULL
354 Chevy NULL NULL
63 Ford 1990 Blue
0 Ford 1990 Red
62 Ford 1990 White
125 Ford 1990 NULL
55 Ford 1991 Blue
0 Ford 1991 Red
9 Ford 1991 White
64 Ford 1991 NULL
39 Ford 1992 Blue
0 Ford 1992 Red
62 Ford 1992 White
101 Ford 1992 NULL
290 Ford NULL NULL
644 NULL NULL NULL
63 NULL 1990 Blue
0 NULL 1990 Red
62 NULL 1990 White
125 NULL 1990 NULL
104 NULL 1991 Blue
54 NULL 1991 Red
104 NULL 1991 White
262 NULL 1991 NULL
110 NULL 1992 Blue
31 NULL 1992 Red
116 NULL 1992 White
257 NULL 1992 NULL
120 Chevy NULL Blue
157 Ford NULL Blue
This document is created with the unregistered version of CHM2PDF Pilot

277 NULL NULL Blue

85 Chevy NULL Red
0 Ford NULL Red
85 NULL NULL Red
149 Chevy NULL White
133 Ford NULL White
282 NULL NULL White

We can then define yet another view on top to generate the "ALL" and "????" placeholders. No NULL data exists in
this example, so I'll look at only the "ALL" case:
CREATE VIEW auto_cube (units, model, year, color) AS
SELECT SUM(units_sold),
ISNULL(model, 'ALL'), ISNULL(CONVERT(char(4), year), 'ALL'),
ISNULL(color, 'ALL')
FROM fullcube
GROUP BY model, year, color WITH CUBE

One advantage of filling out the cube with placeholder rows is that no matter what combination you formulate, you
get an answer. There are no unknowns and there's no need to use the GROUPING function. Had we not constructed
the cube, this query would return no rows found instead of a row with 0 units:
SELECT * FROM auto_cube WHERE model='Chevy' AND color='ALL'
AND year='1990'

units model year color

----- ----- ---- -----
0 Chevy 1990 ALL

Just for fun, here's the equivalent of the fullcube view as a derived table:
SELECT units_sold=SUM(T1.units_sold), T1.model, T1.year, T1.color
FROM (
SELECT D.units_sold, D.model, D.year, D.color
FROM automobile_sales_detail D
UNION ALL
SELECT DISTINCT 0,
A.model, C.year, B.color
FROM
automobile_sales_detail A
CROSS JOIN
automobile_sales_detail B
CROSS JOIN
automobile_sales_detail C
)
AS T1
GROUP BY T1.model, T1.year, T1.color WITH CUBE
GO

Now that you've seen these somewhat exotic uses of CROSS JOIN and views to fill out the cube, be aware that if
you had wanted the whole cube to be generated, you might have inserted some placeholder rows with a value of 0 in
the base table itself (assuming you had access to do so). That way, every combination would be accounted for in the
base table itself.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
This chapter examined the SELECT statement, the hallmark of the SQL language. Although I presented a fairly
well-rounded treatment of SELECT, instead of focusing on basic queries I looked at more subtle issues and SQL
Server-specific capabilities.

We discussed the issues of NULL and three-valued logic, and you learned why you must always consider these
issues when you're formulating queries. In three-valued logic, an answer is TRUE, FALSE, or Unknown. If you must
work with NULL values but you don't understand three-valued logic, you'll introduce bugs. You must fully understand
three-valued logic or structure the database to avoid the use of NULL.

You must also understand NULL for doing JOIN operations, and I discussed issues concerning OUTER JOIN with
several examples. Unlike a JOIN based on equality, the JOIN order for an OUTER JOIN is vitally important. We
looked at search expressions such as LIKE and the issues of trailing blanks. We examined aggregate functions, again
with a discussion of how you must understand NULL for proper use of these functions.

Finally, we looked at the CUBE and ROLLUP extensions to the standard GROUP BY clause for use with aggregate
functions, and we discussed some tips and techniques for using these capabilities.

Although this chapter is far from a complete treatise on the SQL language or SQL Server's capabilities, it
demonstrates how to use SQL Server facilities and extensions to help you solve real-world problems and write better
applications.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 8

Indexes
Indexes are the other significant user-defined, on-disk data structure besides tables. An index provides fast access to
data when the data can be searched by the value that is the index key. You learned some basic information about
indexes in Chapter 3, but to really understand the benefit that indexes can provide and how to determine the best
indexes for your environment, you need to take a deeper look into the organization of SQL Server indexes.

In this chapter, I'll show you the physical organization of index pages for both types of SQL Server indexes, clustered
and nonclustered. I'll discuss the various options available when you create and re-create indexes, and I'll tell you
how, when, and why to rebuild your indexes. I'll tell you about SQL Server 2000's online index defragmentation utility
and about a tool for determining whether your indexes need defragmenting.

Indexes allow data to be organized in a way that allows optimum performance when you access or modify it. SQL
Server does not need indexes to successfully retrieve results for your SELECT statements or to find and modify the
specified rows in your data modification statements. Most of the examples you saw in Chapter 6 could be executed
just fine without any indexes at all. However, as your tables get larger, the value of using proper indexes becomes
obvious. You can use indexes to quickly find data rows that satisfy conditions in your WHERE clauses, to find
matching rows in your JOIN clauses, or to efficiently maintain uniqueness of your key columns during INSERT and
UPDATE operations. In some cases, you can use indexes to help SQL Server sort, aggregate, or group your data or
to find the first few rows as indicated in a TOP clause.

It is the job of the query optimizer to determine which indexes, if any, are most useful in processing a specific query.
The final choice of which indexes to use is one of the most important components of the query optimizer's execution
plan. I'll tell you a lot more about the query optimizer in Chapters 15 and 16, including the factors that determine
whether the query optimizer actually chooses to use indexes at all. In this chapter, I'll focus on what indexes look like
and how they can speed up your queries.
This document is created with the unregistered version of CHM2PDF Pilot

34

Index Organization
Think of the indexes that you see in your everyday life—those in books and other documents. Suppose you're trying
to write a SELECT statement in SQL Server using the CASE expression, and you're using two SQL Server
documents to find out how to write the statement. One document is the Microsoft SQL Server 2000 Transact-SQL
Language Reference Manual. The other is this book, Inside Microsoft SQL Server 2000. You can quickly find
information in either book about CASE, even though the two books are organized differently.

In the T-SQL language reference, all the information is organized alphabetically. You know that CASE will be near
the front, so you can just ignore the last 80 percent of the book. Keywords are shown at the top of each page to tell
you what topics are on that page. So you can quickly flip through just a few pages and end up at a page that has
BREAK as the first keyword and CEILING as the last keyword, and you know that CASE will be on this page. If
CASE is not on this page, it's not in the book. And once you find the entry for CASE, right between BULK INSERT
and CAST, you'll find all the information you need, including lots of helpful examples.

Next, you try to find CASE in this book. There are no helpful key words at the top of each page, but there's an index
at the back of the book and all the index entries are organized alphabetically. So again, you can make use of the fact
that CASE is near the front of the alphabet and quickly find it between "cascading updates" and "case-insensitive
searches." However, unlike in the reference manual, once you find the word CASE, there are no nice neat examples
right in front of you. Instead, the index gives you pointers. It tells you what pages to look at—it might list two or three
pages that point you to various places in the book. If you look up SELECT in the back of the book, however, there
might be dozens of pages listed, and if you look up SQL Server, there might be hundreds.

These searches through the two books are analogous to using clustered and nonclustered indexes. In a clustered
index, the data is actually stored in order, just as the reference manual has all the main topics in order. Once you find
the data you're looking for, you're done with the search. In a nonclustered index, the index is a completely separate
structure from the data itself. Once you find what you're looking for in the index, you have to follow pointers to the
actual data. A nonclustered index in SQL Server is very much like the index in the back of this book. If there is only
one page reference listed in the index, you can quickly find the information. If a dozen pages are listed, it's quite a bit
slower. If hundreds of pages are listed, you might think there's no point in using the index at all. Unless you can
narrow down your topic of interest to something more specific, the index might not be of much help.

In Chapter 3, I explained that both clustered and nonclustered indexes in SQL Server store their information using
standard B-trees, as shown in Figure 8-1. A B-tree provides fast access to data by searching on a key value of the
index. B-trees cluster records with similar keys. The B stands for balanced, and balancing the tree is a core feature of
a B-tree's usefulness. The trees are managed, and branches are grafted as necessary, so that navigating down the tree
to find a value and locate a specific record takes only a few page accesses. Because the trees are balanced, finding
any record requires about the same amount of resources, and retrieval speed is consistent because the index has the
same depth throughout.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 8-1. A B-tree for a SQL Server index.

An index consists of a tree with a root from which the navigation begins, possible intermediate index levels, and
bottom-level leaf pages. You use the index to find the correct leaf page. The number of levels in an index will vary
depending on the number of rows in the table and the size of the key column or columns for the index. If you create an
index using a large key, fewer entries will fit on a page, so more pages (and possibly more levels) will be needed for
the index. On a qualified select, update, or delete, the correct leaf page will be the lowest page of the tree in which
one or more rows with the specified key or keys reside. A qualified operation is one that affects only specific rows
that satisfy the conditions of a WHERE clause, as opposed to accessing the whole table. In any index, whether
clustered or nonclustered, the leaf level contains every key value, in key sequence.

Clustered Indexes
The leaf level of a clustered index contains the data pages, not just the index keys. Another way to say this is that the
data itself is part of the clustered index. A clustered index keeps the data in a table ordered around the key. The data
pages in the table are kept in a doubly linked list called the page chain. The order of pages in the page chain, and the
order of rows on the data pages, is the order of the index key or keys. Deciding which key to cluster on is an
important performance consideration. When the index is traversed to the leaf level, the data itself has been retrieved,
not simply pointed to.

Because the actual page chain for the data pages can be ordered in only one way, a table can have only one
clustered index. The query optimizer strongly favors a clustered index because such an index allows the data to be
found directly at the leaf level. Because it defines the actual order of the data, a clustered index allows especially fast
access for queries looking for a range of values. The query optimizer detects that only a certain range of data pages
must be scanned.

Most tables should have a clustered index. If your table will have only one index, it generally should be clustered.
Many documents describing SQL Server indexes will tell you that the clustered index physically stores the data in
sorted order. This can be misleading if you think of physical storage as the disk itself. If a clustered index had to keep
the data on the actual disk in a particular order, it could be prohibitively expensive to make changes. If a page got too
full and had to be split in two, all the data on all the succeeding pages would have to be moved down. Sorted order in
a clustered index simply means that the data page chain is in order. If SQL Server follows the page chain, it can
access each row in clustered index order, but new pages can be added by simply adjusting the links in the page chain.
I'll tell you more about page splitting and moving rows in Chapter 9 when I discuss data modification.

In SQL Server 2000, all clustered indexes are unique. If you build a clustered index without specifying the unique
keyword, SQL Server forces uniqueness by adding a uniqueifier to the rows when necessary. This uniqueifier is a
4-byte value added as an additional sort key to only the rows that have duplicates of their primary sort key. You'll be
able to see this extra value when we look at the actual structure of index rows later in this chapter.
This document is created with the unregistered version of CHM2PDF Pilot

Nonclustered Indexes
In a nonclustered index, the lowest level of the tree (the leaf level) contains a bookmark that tells SQL Server where
to find the data row corresponding to the key in the index. As you saw in Chapter 3, a bookmark can take one of
two forms. If the table has a clustered index, the bookmark is the clustered index key for the corresponding data row.
If the table is a heap (in other words, it has no clustered index), the bookmark is a RID, which is an actual row
locator in the form File#:Page#:Slot#. (In contrast, in a clustered index, the leaf page is the data page.)

The presence or absence of a nonclustered index doesn't affect how the data pages are organized, so you're not
restricted to having only one nonclustered index per table, as is the case with clustered indexes. Each table can include
as many as 249 nonclustered indexes, but you'll usually want to have far fewer than that.

When you search for data using a nonclustered index, the index is traversed and then SQL Server retrieves the
record or records pointed to by the leaf-level indexes. For example, if you're looking for a data page using an index
with a depth of three—a root page, one intermediate page, and the leaf page—all three index pages must be
traversed. If the leaf level contains a clustered index key, all the levels of the clustered index then have to be traversed
to locate the specific row. The clustered index will probably also have three levels, but in this case remember that the
leaf level is the data itself. There are two additional index levels separate from the data, typically one less than the
number of levels needed for a nonclustered index. The data page still must be retrieved, but because it has been
exactly identified there's no need to scan the entire table. Still, it takes six logical I/O operations to get one data page.
You can see that a nonclustered index is a win only if it's highly selective.

Figure 8-2 illustrates this process without showing you the individual levels of the B-trees. I want to find the first name
for the employee named Anson, and I have a nonclustered index on the last name and a clustered index on the
employee ID. The nonclustered index uses the clustered keys as its bookmarks. Searching the index for Anson, SQL
Server finds that the associated clustered index key is 7. It then traverses the clustered index looking for the row with
a key of 7, and it finds Kim as the first name in the row I'm looking for.

Figure 8-2. SQL Server traverses both the clustered and nonclustered indexes to find the first name for the
employee named Anson.
This document is created with the unregistered version of CHM2PDF Pilot

34

Creating an Index
The typical syntax for creating an index is straightforward:

CREATE [UNIQUE] [CLUSTERED | NONCLUSTERED] INDEX index_name

ON table_name (column_name [ASC | DESC][,...n])

When you create an index, you must specify a name for it. You must also specify the table on which the index will be
built, and then one or more columns. For each column, you can specify that the leaf level will store the key values
sorted in either ascending (ASC) or descending (DESC) order. The default is ascending. You can specify that SQL
Server must enforce uniqueness of the key values by using the keyword UNIQUE. If you don't specify UNIQUE,
duplicate key values will be allowed. You can specify that the index be either clustered or nonclustered. Nonclustered
is the default.

CREATE INDEX has some additional options available for specialized purposes. You can add a WITH clause to
the CREATE INDEX command:
[WITH
[FILLFACTOR = fillfactor]
[[,] [PAD_INDEX]
[[,] IGNORE_DUP_KEY]
[[,] DROP_EXISTING]
[[,] STATISTICS_NORECOMPUTE]
[[,] SORT_IN_TEMPDB]
]

FILLFACTOR is probably the most commonly used of these options. FILLFACTOR lets you reserve some space
on each leaf page of an index. In a clustered index, since the leaf level contains the data, you can use FILLFACTOR
to control how much space to leave in the table itself. By reserving free space, you can later avoid the need to split
pages to make room for a new entry. Chapter 3 includes some discussion of index management and page splitting,
and you'll find more details in Chapter 9. But remember that FILLFACTOR is not maintained; it indicates only how
much space is reserved with the existing data at the time the index is built. If you need to, you can use the DBCC
DBREINDEX command to rebuild the index and reestablish the original FILLFACTOR specified.
TIP
If you plan to rebuild all of a table's indexes, simply specify the clustered index with DBCC DBREINDEX. Doing so
internally rebuilds the entire table and all nonclustered indexes.

FILLFACTOR isn't usually specified on an index-by-index basis, but you can specify it this way for fine-tuning. If
FILLFACTOR isn't specified, the serverwide default is used. The value is set for the server via sp_configure,
fillfactor. This value is 0 by default, which means that leaf pages of indexes are made as full as possible.
FILLFACTOR generally applies only to the index's leaf page (the data page for a clustered index). In specialized and
high-use situations, you might want to reserve space in the intermediate index pages to avoid page splits there, too.
You can do this by specifying the PAD_INDEX option, which uses the same value as FILLFACTOR.

The DROP_EXISTING option specifies that a given index should be dropped and rebuilt as a single transaction.
This option is particularly useful when you rebuild clustered indexes. Normally, when a clustered index is dropped,
every nonclustered index has to be rebuilt to change its bookmarks to RIDs instead of the clustering keys. Then, if a
clustered index is built (or rebuilt), all the nonclustered indexes need to be rebuilt again to update the bookmarks. The
DROP_EXISTING option of the CREATE INDEX command allows a clustered index to be rebuilt without having
This document is created with the unregistered version of CHM2PDF Pilot

to rebuild the nonclustered indexes twice. If you are creating the index on the exact same keys that it had previously,
the nonclustered indexes do not need to be rebuilt at all. If you are changing the key definition, the nonclustered
indexes are rebuilt only once, after the clustered index is rebuilt.

You can ensure the uniqueness of an index key by defining the index as UNIQUE or by defining a PRIMARY KEY
or UNIQUE constraint. If an UPDATE or INSERT statement would affect multiple rows, and if even one row is
found that would cause duplicate keys defined as unique, the entire statement is aborted and no rows are affected.
Alternatively, when you create the unique index, you can use the IGNORE_DUP_KEY option so that a duplicate key
error on a multiple-row INSERT won't cause the entire statement to be rolled back. The nonunique row will be
discarded, and all other rows will be inserted or updated. IGNORE_DUP_KEY doesn't allow the uniqueness of the
index to be violated; instead, it makes a violation in a multiple-row data modification nonfatal to all the nonviolating
rows.

I'll discuss the STATISTICS_NORECOMPUTE option in Chapter 15, when I discuss statistics maintenance.

The SORT_IN_TEMPDB option allows you to control where SQL Server performs the sort operation on the key
values needed to build an index. The default is that SQL Server uses space from the filegroup on which the index is to
be created. While the index is being built, SQL Server scans the data pages to find the key values and then builds
leaf-level index rows in internal sort buffers. When these sort buffers are filled, they are written to disk. The disk
heads for the database can then move back and forth between the base table pages and the work area where the sort
buffers are being stored. If, instead, your CREATE INDEX command includes the option SORT_IN_TEMPDB,
performance can be greatly improved, particularly if your tempdb database is on a separate physical disk from the
database you're working with, with its own controller. You can optimize head movement because two separate heads
read the base table pages and manage the sort buffers. You can get even more improvement in index creation speed if
your tempdb is on a faster disk than your user database and you use the SORT_IN_TEMPDB option. As an
alternative to using the SORT_IN_TEMPDB option, you can create separate filegroups for a table and its indexes.
(That is, the table is on one filegroup and its indexes are on another.) If the two filegroups are on different disks with
their own controllers, you can also minimize the disk head movement.

Constraints and Indexes

As I mentioned in Chapter 6, when you declare a PRIMARY KEY or UNIQUE constraint, a unique index is
created on one or more columns, just as if you had used the CREATE INDEX command. The names of indexes that
are built to support these constraints are the same as the constraint names. In terms of internal storage and
maintenance of indexes, there is no difference between unique indexes created using the CREATE INDEX command
and indexes created to support constraints. The query optimizer makes decisions based on the presence of the unique
index rather than on the fact that a column was declared as a primary key. How the index got there in the first place is
irrelevant to the query optimizer.

When you create a table that includes PRIMARY KEY or UNIQUE constraints, you can specify whether the
associated index will be clustered or nonclustered, and you can also specify the fillfactor. Since the fillfactor applies
only at the time the index is created, and since there is no data when you first create the table, it might seem that
specifying the fillfactor at that time is completely useless. However, if after the table is populated you decide to use
DBCC DBREINDEX to rebuild all your indexes, you can specify a fillfactor of 0 to indicate that SQL Server should
use the fillfactor that was specified when the index was created. You can also specify a fillfactor when you use
ALTER TABLE to add a PRIMARY KEY or UNIQUE constraint to a table, and if the table already had data in it,
the fillfactor value is applied when you build the index to support the new constraint.

If you check the documentation for CREATE TABLE and ALTER TABLE, you'll see that the
SORT_IN_TEMPDB option is not available for either command. It really doesn't make sense to specify a sort
location when you first create the table because there's nothing to sort. However, the fact that you can't specify this
alternate location when you add a PRIMARY KEY or UNIQUE constraint to a table with existing data seems like an
oversight. Also note that SORT_IN_TEMPDB is not an option when you use DBCC DBREINDEX. Again, there's
no reason why it couldn't have been included, but it isn't available in this release.
This document is created with the unregistered version of CHM2PDF Pilot

The biggest difference between indexes created using the CREATE INDEX command and indexes that support
constraints is in how you can drop the index. The DROP INDEX command allows you to drop only indexes that
were built with the CREATE INDEX command. To drop indexes that support constraints, you must use ALTER
TABLE to drop the constraint. In addition, to drop a PRIMARY KEY or UNIQUE constraint that has any
FOREIGN KEY constraints referencing it, you must first drop the FOREIGN KEY constraint. This can leave you
with a window of vulnerability while you redefine your constraints and rebuild your indexes. While the FOREIGN
KEY constraint is gone, an INSERT statement can add a row to the table that violates your referential integrity.

One way to avoid this problem is to use DBCC DBREINDEX, which drops and rebuilds all your indexes on a table
in a single transaction, without requiring the auxiliary step of removing FOREIGN KEY constraints. Alternatively, you
can use the CREATE INDEX command with the DROP_EXISTING option. In most cases, you cannot use this
command on an index that supports a constraint. However, there is an exception. For example, the following
command attempts to rebuild the index on the title_id column of the titles table in the pubs database:
CREATE CLUSTERED INDEX UPKCL_titleidind ON titles(title_id)
WITH DROP_EXISTING

SQL Server returned this error message to me:

Server: Msg 1907, Level 16, State 1, Line 1
Cannot re-create index 'UPKCL_titleidind'. The new index definition
does not match the constraint being enforced by the existing index.

How could I have known that this index supported a constraint? First of all, the name includes UPKCL, which is a
big clue. However, the output of sp_helpindex tells us only the names of the indexes, the property (clustered or
unique), and the columns the index is on. It doesn't tell us if the index supports a constraint. However, if we execute
sp_help on a table, the output will tell us that UPKCL_titleidind is a PRIMARY KEY constraint. The error message
indicates that we can't use the DROP_EXISTING clause to rebuild this index because the new definition doesn't
match the current index. We can use this command as long as the properties of the new index are exactly the same as
the old. In this case, I didn't specify that the index was to be UNIQUE. We can rephrase the command as follows so
that the CREATE INDEX is successful:
CREATE UNIQUE CLUSTERED INDEX UPKCL_titleidind ON titles(title_id)
WITH DROP_EXISTING
This document is created with the unregistered version of CHM2PDF Pilot

34

The Structure of Index Pages

Index pages are structured much like data pages. As with all other types of pages in SQL Server, index pages have a
fixed size of 8 KB, or 8192 bytes. Index pages also have a 96-byte header, but just like in data pages, there is an
offset array at the end of the page with two bytes for each row to indicate the offset of that row on the page.
However, if you use DBCC PAGE with style 1 to print out the individual rows on an index page, the slot array is not
shown. If you use style 2, it will only print out all the bytes on a page with no attempt to separate the bytes into
individual rows. You can then look at the bottom of the page and notice that each set of two bytes does refer to a
byte offset on the page. Each index has a row in the sysindexes table, with an indid value of either 1, for a clustered
index, or a number between 2 and 250, indicating a nonclustered index. An indid value of 255 indicates LOB data (
text, ntext or image information). The root column value contains a file number and page number where the root of
the index can be found. You can then use DBCC PAGE to examine index pages, just as you do for data pages.

The header information for index pages is almost identical to the header information for data pages. The only
difference is the value for type, which is 1 for data and 2 for index. The header of an index page also has nonzero
values for level and indexId. For data pages, these values are both always 0.

There are basically three different kinds of index pages: leaf level for nonclustered indexes, node (nonleaf) level for
clustered indexes, and node level for nonclustered indexes. There isn't really a separate structure for leaf level pages
of a clustered index because those are the data pages, which we've already seen in detail. There is, however, one
special case for leaf-level clustered index pages.

Clustered Index Rows with a Uniqueifier

If your clustered index was not created with the UNIQUE property, SQL Server adds a 4-byte field when necessary
to make each key unique. Since clustered index keys are used as bookmarks to identify the base rows being
referenced by nonclustered indexes, there needs to be a unique way to refer to each row in a clustered index. SQL
Server adds the uniquifier only when necessary—that is, when duplicate keys are added to the table. As an example,
I'll use the same table I initially used in Chapter 6 to illustrate the row structure of a table with all fixed-length columns.
I'll create an identical table with a different name, then I'll add a clustered (nonunique) index to the table:
USE pubs
GO
CREATE TABLE Clustered_Dupes
(Col1 char(5) NOT NULL,
Col2 int NOT NULL,
Col3 char(3) NULL,
Col4 char(6) NOT NULL,
Col5 float NOT NULL)
GO
CREATE CLUSTERED INDEX Cl_dupes_col1 ON Clustered_Dupes(col1)

The rows in syscolumns look identical to the ones for the table in Chapter 6, without the clustered index. However, if
you look at the sysindexes row for this table, you'll notice something different.
SELECT first, indid, keycnt, name FROM sysindexes
WHERE id = object_id ('Clustered_Dupes')
RESULT:
first indid keycnt name
-------------- ------ ------ ---------------
0x000000000000 1 2 Cl_dupes_col1
This document is created with the unregistered version of CHM2PDF Pilot

The column called keycnt, which indicates the number of keys an index has, is 2. If I had created this index using the
UNIQUE qualifier, the keycnt value would be 1. If I had looked at the sysindexes row before adding a clustered
index, when the table was still a heap, the row for the table would have had a keycnt value of 0. I'll now add the same
initial row that I added to the table in Chapter 6, and then I'll look at sysindexes to find the first page of the table:
INSERT Clustered_Dupes VALUES ('ABCDE', 123, null, 'CCCC', 4567.8)
GO
SELECT first, root, id,indid FROM sysindexes
WHERE id = object_id('Clustered_Dupes')

RESULT:
first root id indid
-------------- -------------- ----------- ------
0xB50500000100 0xB30500000100 1426104121 1

The first column tells me that the first page of the table is 1461 (0x05B5), so I can use DBCC PAGE to examine
that page. Remember to turn on trace flag 3604 prior to executing this undocumented DBCC command.
DBCC TRACEON (3604)
GO
DBCC PAGE (5,1,1461, 1)

The only row on the page looks exactly as it was shown in Figure 6-7, but I'll reproduce it again here, in Figure 8-3.
When you read the row output from DBCC PAGE, remember that each two displayed characters represents a byte
and that the bytes are displayed from right to left within each group of 4 bytes in the output. For character fields, you
can just treat each byte as an ASCII code and convert that to the associated character. Numeric fields are stored
with the low-order byte first, so within each numeric field, we must swap bytes to determine what value is being
stored. Right now, there are no duplicates of the clustered key, so no extra information has to be provided. However,
if I add two additional rows, with duplicate values in col1, the row structure changes:
INSERT Clustered_Dupes VALUES ('ABCDE', 456, null, 'DDDD', 4567.8)
INSERT Clustered_Dupes VALUES ('ABCDE', 64, null, 'EEEE', 4567.8)

Figure 8-4 shows the three rows that are now on the page.

Figure 8-3. A data row containing all fixed-length columns, and a unique value in the clustered key column.
This document is created with the unregistered version of CHM2PDF Pilot

The first difference in the second two rows is that the bits in the first byte (TagA) are different. Bit 5 is on, giving
TagA a value of 0x30, which means the variable block is present in the row. Without this bit on, TagA would have a
value of 0x10. The "extra" variable-length portions of the second two rows are shaded in the figure. You can see that
8 extra bytes are added when we have a duplicate row. In this case, the first 4 extra bytes are added because the
uniqueifier is considered a variable-length column. Since there were no variable-length columns before, SQL Server
adds 2 bytes to indicate the number of variable-length columns present. These bytes are at offsets 33-34 in these
rows with the duplicate keys and have the value of 1. The next 2 bytes, (offsets 35-36) indicate the position where the
first variable length column ends. In both these rows, the value is 0x29, which converts to 41 decimal. The last 4 bytes
(offsets 37-40) are the actual uniqueifier. In the second row, which has the first duplicate, the uniquifier is 1. The third
row has a uniquifier of 2.

Figure 8-4. Three data rows containing duplicate values in the clustered key column.

Index Row Formats

The row structure of an index entry is very similar to the structure of a data row. An index row does not use the
TagB or Fsize row header values. In place of the Fsize field, which indicates where the fixed-length portion of a row
ends, the page header pminlen value is used to decode an index row. The pminlen value indicates the offset at which
the fixed-length data portion of the row ends. If there are no variable-length or nullable columns in the index row, that
is the end of the row. Only if the index row has nullable columns are the field called Ncol and the null bitmap both
present. The Ncol field contains a value indicating how many columns are in the index row; this value is needed to
determine how many bits are in the null bitmap. Data rows have an Ncol field and null bitmap whether or not any
columns allow NULL, but index rows have only a null bitmap and an Ncol field if any NULLs are allowed in an index
column. Table 8-1 shows the general format of an index row.

Table 8-1. Information stored in an index row.

Information Mnemonic Size
This document is created with the unregistered version of CHM2PDF Pilot

TagA

Some of the relevant bits are:

 Bits 1 through 3: Taken as a 3-bit
value, 0 indicates a primary record, 3
indicates an index record, and 5
Status Bits A 1 byte
indicates a ghost index record. (I'll
discuss ghost records in Chapter 9.)
 Bit 4: Indicates that a NULL bitmap
exists.
 Bit 5: Indicates that variable-length
columns exist in the row.
Fixed-length data Fdata pminlen - 1
Number of columns Ncol 2 bytes
NULL bitmap (1 bit for each
column in the table; a 1 indicates
Nullbits Ceiling (Ncol / 8)
that the corresponding column is
NULL)
Number of variable-length
VarCount 2 bytes
columns; only present if >0
Variable column offset array; only
VarOffset 2 * VarCount
present if VarCount > 0
Variable-length data, if any VarData

The data contents of an index row depend on the level of the index. All rows except those in the leaf level contain a
6-byte down-page pointer in addition to the key values and bookmarks. The down-page pointer is the last column in
the fixed-data portion of the row. In nonclustered indexes, the nonclustered keys and bookmarks are treated as
normal columns. They can reside in either the fixed or variable portion of the row, depending on how each of the key
columns was defined. A bookmark that is a RID, however, is always part of the fixed-length data.

If the bookmark is a clustered key value and the clustered and nonclustered indexes share columns, the actual data is
stored only once. For example, if your clustered index key is lastname and you have a nonclustered index on (
firstname, lastname), the index rows will not store the value of lastname twice. I'll show you an example of this
shortly.

Clustered Index Node Rows

The node levels of a clustered index contain pointers to pages at the next level down in the index, along with the first
key value on each page pointed to. Page pointers are 6 bytes: 2 bytes for the file number and 4 bytes for the page
number in the file. The following code creates and populates a table that I'll use to show you the structure of clustered
index node rows.
CREATE TABLE clustered_nodupes (
id int NOT NULL ,
str1 char (5) NOT NULL ,
str2 char (600) NULL
)
GO
CREATE CLUSTERED INDEX idxCL ON clustered_nodupes(str1)
GO
SET NOCOUNT ON
GO
DECLARE @i int
SET @i = 1240
WHILE @i < 13000 BEGIN
This document is created with the unregistered version of CHM2PDF Pilot

INSERT INTO clustered_nodupes

SELECT @i, cast(@i AS char), cast(@i AS char)
SET @i = @i + 1
END
GO
SELECT first, root, id, indid FROM sysindexes
WHERE id = object_id('clustered_nodupes')

RESULT:
first root id indid
-------------- -------------- ----------- ------
0xB80500000100 0xB60500000100 1458104235 1

DBCC TRACEON (3604)

GO
DBCC PAGE (5,1,1462, 1)

The root of this index is at 0x05B6, which is 1462 decimal. Using DBCC PAGE to look at the root, we can see two
index entries, as shown in Figure 8-5. When you examine index pages, you need to be aware that the first index key
entry on each page is frequently either meaningless or empty. The down-page pointer is valid, but the data for the
index key might not be a valid value. When SQL Server traverses the index, it starts looking for a value by comparing
the search key with the second key value on the page. If the value being sought is less than the second entry on the
page, SQL Server follows the page pointer indicated in the first index entry. In this example, the down-page pointer is
at byte offsets 6-9, with a hex value of 0x0839. In decimal, this is 2105.

We can then use DBCC PAGE again to look at page 2105, part of which is shown in Figure 8-6. The row structure
is identical to the rows on page 1462, the root page.

Figure 8-5. The root page of a clustered index.

This document is created with the unregistered version of CHM2PDF Pilot

Figure 8-6. An intermediate index-level page for a clustered index.

The first row has a meaningless key value, but the page-down pointer should be the first page in clustered index
order. The hex value for the page-down pointer is 0x07dd, which is 2013 decimal. That page is the leaf level page,
and has a structure of a normal data page. If we use DBCC PAGE to look at that page, we'll see that the first row
has the value of '10000', which is the minimum value of the str1 column in the table. We can verify this with the
following query:
SELECT min(str1)
FROM clustered_nodupes

Remember that the clustered index column is a char(5) column. Although the first number I entered was 1240, when
converted to a char(5), that's '1240 '. When sorted alphabetically, '10000' comes well before '1240 '.

Nonclustered Index Leaf Rows

Index node rows in a clustered index contain only the first key value of the page they are pointing to in the next level
and page-down pointers to guide SQL Server in traversing the index. Nonclustered index rows can contain much
more information. The rows in the leaf level of a nonclustered index contain every key value and a bookmark. I'll
show you three different examples of nonclustered index leaf rows. First, we'll look at the leaf level of a nonclustered
index built on a heap. I'll use the same code I used to build the clustered index with no duplicates in the previous
example, but the index I build on the str1 column will be nonclustered. I'll also put only a few rows in the table so that
the root page will be the entire index.
This document is created with the unregistered version of CHM2PDF Pilot

CREATE TABLE nc_heap_nodupes (

id int NOT NULL ,
str1 char (5) NOT NULL ,
str2 char (600) NULL
)
GO
CREATE UNIQUE INDEX idxNC_heap ON nc_heap_nodupes (str1)
GO
SET NOCOUNT ON
GO
DECLARE @i int
SET @i = 1240
WHILE @i < 1300 BEGIN
INSERT INTO nc_heap_nodupes
SELECT @i, cast(@i AS char), cast(@i AS char)
SET @i = @i + 1
END
GO
SELECT first, root, id, indid FROM sysindexes
WHERE id = object_id('nc_heap_nodupes')
AND indid > 1

RESULT:
first root id indid
-------------- -------------- ----------- ------
0xC10500000100 0xC10500000100 1506104406 2

DBCC TRACEON (3604)

GO
DBCC PAGE (5,1,1473, 1)

Since this index is built on a heap, the bookmark is an 8-byte RID. As you can see in Figure 8-7, this RID is
fixed-length and is located in the index row immediately after the index key value of '1240 '. The first 4 bytes are the
page address, the next 2 bytes are the file ID, and the last 2 bytes are the slot number.

Figure 8-7. An index row on a leaf-level page from a nonclustered index on a heap.

I'll now build a nonclustered index on a similar table, but before building the nonclustered index, I'll build a clustered
index on a varchar column so we can see what an index looks like when a bookmark is a clustered index key.. Also,
in order not to have the same values in the same sequence in both the str1 and str2 columns, I'll introduce a bit of
randomness into the generation of the values for str2. If you run this script, the randomness might end up generating
some duplicate values for the unique clustered index column str2. Since each row is inserted in its own INSERT
statement, a violation of uniqueness will cause only that one row to be rejected. If you get error messages about
PRIMARY KEY violation, just ignore them. You'll still end up with enough rows.
CREATE TABLE nc_nodupes (
id int NOT NULL ,
This document is created with the unregistered version of CHM2PDF Pilot

str1 char (5) NOT NULL ,

str2 varchar (10) NULL
)
GO
CREATE UNIQUE CLUSTERED INDEX idxcl_str2 on nc_nodupes (str2)
CREATE UNIQUE INDEX idxNC ON nc_nodupes (str1)
GO
SET NOCOUNT ON
GO
DECLARE @i int
SET @i = 1240
WHILE @i < 1300 BEGIN
INSERT INTO nc_nodupes
SELECT @i, cast(@i AS char),
cast(cast(@i * rand() AS int) as char)
SET @i = @i + 1
END
GO
SELECT first, root, id, indid FROM sysindexes
WHERE id = object_id('nc_nodupes')
AND indid > 1

RESULT:
first root id indid
-------------- -------------- ----------- ------
0xD00500000100 0xD00500000100 1586104691 2

DBCC TRACEON (3604)

GO
DBCC PAGE (5,1,1488, 1)

Figure 8-8. An index row on a leaf-level page from a nonclustered index built on a clustered table.

The index row contains the nonclustered key value of '1240 ', and the bookmark is the clustered key '822'. Because
the clustered key is variable length, it comes at the end of the row after the Ncol value and null bitmap. A
variable-length key column in an index uses only the number of bytes needed to store the value inserted. In the script
to create this table, an int is converted to a char to get the value for the field. Since no length is supplied for the char,
SQL Server by default allows 30 characters. However, since the table was defined to allow a maximum of 10
characters in that column, that is all that was stored in the row. If the script to create the table had actually converted
to char(3), the clustered key bookmark in the index row would use only 3 bytes.

The third example will show a composite nonclustered index on str1 and str2 and an overlapping clustered index on
str2. I'll just show you the code to create the table and indexes; the code to populate the table and to find the root of
the index is identical to the code in the previous example.
CREATE TABLE nc_overlap (
id int NOT NULL ,
This document is created with the unregistered version of CHM2PDF Pilot

str1 char (5) NOT NULL ,

str2 char (10) NULL
)
GO
CREATE UNIQUE CLUSTERED INDEX idxCL_overlap ON nc_overlap(str2)
CREATE UNIQUE INDEX idxNC_overlap ON nc_overlap (str1, str2)
GO

Figure 8-9 shows a leaf-level index row from the nonclustered composite index on the table nc_overlap.

Figure 8-9. An index row on a leaf-level page from a composite nonclustered index with an overlapping
clustered key.

Note that the value in column str2 ('375' in this row) is part of the index key for the nonclustered index as well as the
bookmark because it is the clustered index key. Although it serves two purposes in the row, its value is not
duplicated. The value '375' occurs only once, taking the maximum allowed space of 10 bytes. Because this is a
leaf-level row, there is no page-down pointer, and because the index is built on a clustered table, there is no RID,
only the clustered key to be used as a bookmark.

Nonclustered Index Node Rows

You now know that the leaf level of a nonclustered index must have a bookmark because from the leaf level you
want to be able to find the actual row of data. The nonleaf levels of a nonclustered index only need to help us traverse
down to pages at the lower levels. If the nonclustered index is unique, the node rows need to have only the
nonclustered key and the page-down pointer. If the index is not unique, the row contains key values, a down-page
pointer, and a bookmark.

Let's say I need to create a table that will have enough rows for more than a single nonclustered index level. I can use
a script similar to the ones shown previously and change the upper limit of the WHILE loop to 13000. I'll do this
twice: I'll create one table with a nonunique nonclustered index on str1 and a clustered index on str2 and another
table with a unique nonclustered index on str1 and a clustered index on str2. For the purposes of creating the index
rows, SQL Server doesn't care whether the keys in the nonunique index actually contain duplicates. If the index is not
defined to be unique, even if all the values are unique, the nonleaf index rows will contain bookmarks. You can see
this is in the index row in Figure 8-10. Figure 8-11 shows an index row for a nonclustered index row for a unique
index.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 8-10. A nonleaf index row for a nonunique, nonclustered index.

Figure 8-11. A nonleaf index row for a unique, nonclustered index.

As you've seen, once you've found the root page of an index in the sysindexes table, you can decode down-page
pointers in every row to find all the pages that belong to the index. The header information of an index page also has
level, prevPage, and nextPage values, so you can ascertain the order of pages at a particular level of the index. As
much as I enjoy decoding strings of bytes and converting hex values to decimal, I don't mind an occasional shortcut.
You can use an undocumented DBCC command that lists all the pages that belong to a particular index. Keep in mind
that Microsoft does not support this command and that it is not guaranteed to maintain the same behavior from release
to release. The command DBCC IND has three parameters:
DBCC IND ({'dbname' | dbid }, { 'objname' | objid },
{ indid | 0 | -1 | -2 })

The first parameter is the database name or the database ID. The second parameter is an object name or object ID
within the database. The third parameter is a specific index ID or one of the values 0, -1, or -2. The values for this
parameter have the following meanings:
0 Displays the page numbers for all IAMs and data pages.
Displays the page numbers for all IAMs, data pages, and
-1
index pages.
-2 Displays the page numbers for all IAMs.
Displays the page numbers for all IAMs and index pages
Index ID for this index. If the index ID is 1 (meaning the clustered
index), the data pages are also displayed.

If we run this command on the table I created for Figure 8-10, which is called nc_nonunique_big, with a third
This document is created with the unregistered version of CHM2PDF Pilot

parameter of 2 (for the nonclustered index), we get the following results:

DBCC IND(5, nc_nonunique_big, 2)
RESULTS:
NextPageFID Next Prev Prev
Page Page Page Page
PageFID PagePID IAMFID IAMPID IndexLevel FID PID FID PID
------- ------- ------ ------ ---------- ----------- ---- ---- ----
1 1594 NULL NULL 0 0 0 0 0
1 1593 1 1594 1 0 0 0 0
1 1596 1 1594 0 1 4608 0 0
1 1597 1 1594 0 1 1599 1 4611
1 1599 1 1594 0 1 1602 1 1597
1 1602 1 1594 0 1 1604 1 1599
1 1604 1 1594 0 1 1606 1 1602
1 1606 1 1594 0 1 1607 1 1604
1 1607 1 1594 0 1 4584 1 1606
1 4584 1 1594 0 1 4585 1 1607
1 4585 1 1594 0 1 4586 1 4584
1 4586 1 1594 0 1 4587 1 4585
1 4587 1 1594 0 1 4588 1 4586
1 4588 1 1594 0 1 4589 1 4587
1 4589 1 1594 0 1 4590 1 4588
1 4590 1 1594 0 0 0 1 4589
1 4591 1 1594 0 1 4611 1 4610
1 4608 1 1594 0 1 4609 1 1596
1 4609 1 1594 0 1 4610 1 4608
1 4610 1 1594 0 1 4591 1 4609
1 4611 1 1594 0 1 1597 1 4591

The columns in the result set are described in Table 8-2. Note that all page references have the file and page
component conveniently split between two columns, so you don't have to do any conversion.

Table 8-2. Column descriptions for DBCC IND output.

Column Meaning
PageFID Index file ID
PagePID Index page ID
IAMFID File ID of the IAM managing this page
IAMPID Page ID of the IAM managing this page
ObjectID (Not shown in output) Object ID
IndexID (Not shown in output) Index ID
Page type: 1 = Data page, 2 = Index page, 10 = IAM
PageType (Not shown in output)
page
IndexLevel Level of index; 0 is leaf
NextPageFID File ID for next page at this level
NextPagePID Page ID for next page at this level
PrevPageFID File ID for previous page at this level
PrevPagePID Page ID for previous page at this level
This document is created with the unregistered version of CHM2PDF Pilot

34

Index Space Requirements

I've shown you the structure and number of bytes in individual index rows, but you really need to be able to translate
that into overall index size. In general, the size of an index is based on the size of the index keys, which determines
how many index rows can fit on an index page and the number of rows in the table.

B-Tree Size
When we talk about index size, we usually mean the size of the index tree. The clustered index does include the data,
but since you still have the data even if you drop the clustered index, we're usually just interested in how much
additional space the nonleaf levels require. A clustered index's node levels typically take up very little space. You have
one index row for each page of table data, so the number of index pages at the level above the leaf (level 1) is the
bytes available per page divided by the index key row size, and this quotient divided into the number of data pages.
You can do a similar computation to determine the pages at level 2. Once you reach a level with only one page, you
can stop your computations because that level is the root.

For example, consider a table of 10,000 pages and 500,000 rows with a clustered key of a 5-byte fixed-length
character. As you saw in Figure 8-5, the index key row size is 12 bytes, so we can fit 674 (8096 bytes available on a
page / 12 bytes per row) rows on an index page. Since we need index rows to point to all 10,000 pages, we'll need
15 index pages (10000 / 674) at level 1. Now, all these index pages need pointers at level 2, and since one page can
contain 15 index rows to point to these 15 pages, level 2 is the root. If our 10,000-page table also has a 5-byte
fixed-length character nonclustered index, the leaf-level rows (level 0) will be 11 bytes long as they will each contain a
clustered index key (5 bytes), a nonclustered index key (5 bytes) and 1 byte for the status bits. The leaf level will
contain every single nonclustered key value along with the corresponding clustered key values. An index page can
hold 736 index rows at this leaf level. We need 500,000 index rows, one for each row of data, so we need 680 leaf
pages. If this nonclustered index is unique, the index rows at the higher levels will be 12 bytes each, so 674 index
rows will fit per page, and we'll need two pages at level 1, with level 2 as the one root page.

So how big are these indexes compared to the table? For the clustered index, we had 16 index pages for a
10,000-page table, which is less than 1 percent of the size of the table. I frequently use 1 percent as a ballpark
estimate for the space requirement of a clustered index, even though you can see that in this case it's an overly large
estimate. On the other hand, our nonclustered index needed 683 pages for the 10,000-page table, which is about 6
percent additional space. For nonclustered indexes, it is much harder to give a good ballpark figure. My example used
a very short key. Nonclustered index keys are frequently much larger, or even composite, so it's not unusual to have
key sizes of over 100n bytes. In that case, we'd need a lot more leaf level pages, and the total nonclustered index size
could be 30 or 40 percent of the size of the table. I've even seen nonclustered indexes that are as big or bigger than
the table itself. Once you have two or three nonclustered indexes, you need to double the space to support these
indexes. Remember that SQL Server allows you to have up to 249 nonclustered indexes! Disk space is cheap, but is
it that cheap? You still need to plan your indexes carefully.

Actual vs. Estimated Size

The actual space used for tables or indexes, as opposed to the ballpark estimates I just described, is stored in the
sysindexes table. A column called rowcnt contains the number of data rows in the table. A column called dpages
contains, for a clustered index or a heap, the number of data pages used; for all other indexes, dpages contains the
number of index pages used. For LOB data (text, ntext, or image), with an indid value of 255, dpages is 0.
Sysindexes also has a column called reserved, which, for a clustered index or a heap contains the number of data
pages reserved; for a nonclustered index, it contains the number of pages reserved for the index itself; and for LOB
This document is created with the unregistered version of CHM2PDF Pilot

data, reserved is a count of the pages allocated. I'll tell you more about what "reserved" means later in this section.

Finally, there is a column in sysindexes called used. For clustered indexes or heaps, it contains the total number of
pages used for all index and table data. For LOB data, used is the number of LOB pages. For nonclustered indexes,
used is the count of index pages. Because no specific value is just the number of nonleaf, or nondata, pages in a
clustered index, you must do additional computations to determine the total number of index pages for a table, not
including the data.

The stored procedure sp_spaceused examines these values and reports the total size used by a table. Keep in mind
that these values in sysindexes are not updated every time a table or index is modified. In particular, immediately after
you create a table and after some bulk operations, sp_spaceused might not accurately reflect the total space because
the values in sysindexes are not accurate.

You can force SQL Server to update the sysindexes data in two ways. The simplest way to force the space used
values to be updated is to ask for this when you execute the sp_spaceused procedure. The procedure takes two
optional parameters: the first is called @objname and is a table name in the current database, and the second is
called @updateusage, which can have a value of TRUE or FALSE. If you specify TRUE, the data in sysindexes will
be updated. The default is FALSE.

Here's an example of forcing SQL Server to update the space used data for a table called charge:
exec sp_spaceused charge, true

Here's the result:

name rows reserved data index_size unused
-------- --------- ------------ ----------- ------------ ---------
charge 100000 10328 KB 5384 KB 4696 KB 248 KB

Remember that after the first 8 pages are allocated to a table, SQL Server grants all future allocations in units of 8
pages, called extents. The sum of all the allocations is the value of reserved. It might be that only a few of the 8 pages
in some of the extents actually have data in them and are counted as used, so that the reserved value is frequently
higher than the used value. The data value is computed by looking at the data column in sysindexes for the clustered
index or heap and adding any LOB pages. The index_size value is computed by taking the used value from
sysindexes for the clustered index or heap, which includes all index pages for all indexes. Because this value includes
the number of data pages, which are at the leaf level of the clustered index, we must subtract the number of data
pages from used pages to get the total number of index pages. The procedure sp_spaceused does all of its space
reporting in KB, which we can divide by 8 to get the number of pages. The unused value is then the leftover amount
after we subtract the data value and the index_size value from the reserved value.

You can also use the DBCC UPDATEUSAGE command to update the space information for every table and index
in sysindexes. Or, by supplying additional arguments, you can have the values updated for only a single table, view, or
index. In addition, DBCC UPDATEUSAGE gives you a report of the changes it made. You might want to limit the
command to only one table because the command works by actually counting all the rows and all the pages in each
table it's interested in. For a huge database, this can take a lot of time. In fact, sp_spaceused with the @updateusage
argument set to TRUE actually calls DBCC UPDATEUSAGE and suppresses the report of changes made. The
following command updates the space used information for every table in the current database:
DBCC UPDATEUSAGE (0)

Here's some of the output I received when I ran this in the Northwind database:
DBCC UPDATEUSAGE: sysindexes row updated for table 'syscolumns' (index ID 2):
USED pages: Changed from (2) to (5) pages.
This document is created with the unregistered version of CHM2PDF Pilot

RSVD pages: Changed from (2) to (5) pages.

DBCC UPDATEUSAGE: sysindexes row updated for table 'Orders' (index ID 2):
USED pages: Changed from (2) to (5) pages.
RSVD pages: Changed from (2) to (5) pages.
DBCC UPDATEUSAGE: sysindexes row updated for table 'Orders' (index ID 3):
USED pages: Changed from (2) to (5) pages.
RSVD pages: Changed from (2) to (5) pages.

DBCC UPDATEUSAGE: sysindexes row updated for table

'Order Details' (index ID 3):
USED pages: Changed from (2) to (6) pages.
RSVD pages: Changed from (2) to (6) pages.

Note that you can use this command to update both the used and reserved values in sysindexes.

It's great to have a way to see how much space is being used by a particular table or index, but what if we're getting
ready to build a database and we want to know how much space our data and indexes will need? For planning
purposes, it would be nice to know this information ahead of time.

If you've already created your tables and indexes, even if they're empty, SQL Server can get column and datatype
information from the syscolumns system table, and it should be able to see any indexes you've defined in the
sysindexes table. There shouldn't be a need for you to type all this information into a spreadsheet. The CD also
includes a set of stored procedures you can use to calculate estimated space requirements when the table and indexes
have already been built. The main procedure is called sp_EstTableSize, and it requires only two parameters: the table
name and the anticipated number of rows. The procedure calculates the storage requirements for the table and all
indexes by extracting information from the sysindexes, syscolumns, and systypes tables. The result is only an estimate
when you have variable-length fields. The procedure has no way of knowing whether a variable-length field will be
completely filled, half full, or mostly empty in every row, so it assumes that variable-length columns will be filled to the
maximum size. If you know that this won't be the case with your data, you can create a second table that more closely
matches the expected data. For example, if you have a varchar(1000) field that you must set to the maximum
because a few rare entries might need it but 99 percent of your rows will use only about 100 bytes, you can create a
second table with a varchar(100) field and run sp_EstTableSize on that table.
NOTE
The procedures for calculating estimated storage space were developed primarily by SQL Server MVP BP Margolin
and are included on the CD with his permission.
This document is created with the unregistered version of CHM2PDF Pilot

34

Managing an Index
SQL Server maintains your indexes automatically. As you add new rows, it automatically inserts them into the correct
position in a table with a clustered index, and it adds new leaf-level rows to your nonclustered indexes that will point
to the new rows. When you remove rows, SQL Server automatically deletes the corresponding leaf-level rows from
your nonclustered indexes. In Chapter 9, you'll see some specific examples of the changes that take place within an
index as data modification operations take place.

Types of Fragmentation
In general, you need to do very little in terms of index maintenance. However, indexes can become fragmented. The
fragmentation can be of two types. Internal fragmentation occurs when space is available within your index
pages—that is, when the indexes are not making the most efficient use of space. External fragmentation occurs when
the logical order of pages does not match the physical order, or when the extents belonging to a table are not
contiguous.

Internal fragmentation means that the index is taking up more space than it needs to. Scanning the entire table involves
more read operations than if no free space were available on your pages. However, internal fragmentation is
sometimes desirable—in fact, you can request internal fragmentation by specifying a low fillfactor value when you
create an index. Having room on a page means that there is space to insert more rows without having to split a page.
Splitting is a relatively expensive operation and can lead to external fragmentation because when a page is split, a new
page must be linked into the indexes page chain, and usually the new page is not contiguous to the page being split.

External fragmentation is truly bad only when SQL Server is doing an ordered scan of all or part of a table or index.
If you're seeking individual rows through an index, it doesn't matter where those rows are physically located—SQL
Server can find them easily. If SQL Server is doing an unordered scan, it can use the IAM pages to determine which
extents need to be fetched, and the IAM pages list the extents in disk order, so the fetching can be very efficient. Only
if the pages need to be fetched in logical order, according to their index key values, do you need to follow the page
chain. If the pages are heavily fragmented, this operation is more expensive than if there were no fragmentation.

Detecting Fragmentation
You can use the DBCC SHOWCONTIG command to report on the fragmentation of an index. Here's the syntax:
DBCC SHOWCONTIG
[ ( { table_name | table_id | view_name | view_id }
[ , index_name | index_id ]
)
]
[ WITH { ALL_INDEXES
| FAST [ , ALL_INDEXES ]
| TABLERESULTS [ , { ALL_INDEXES } ]
[ , { FAST | ALL_LEVELS } ]
}
]

You can ask to see the information for all the indexes on a table or view or just one index, and you can specify the
object and index by name or by ID number. Alternatively, you can use the ALL_INDEXES option, which provides
an individual report for each index on the object, regardless of whether a specific index is specified.
This document is created with the unregistered version of CHM2PDF Pilot

Here's some sample output from running a basic DBCC SHOWCONTIG on the order details table in the
Northwind database:
DBCC SHOWCONTIG scanning 'Order Details' table...
Table: 'Order Details' (325576198); index ID: 1, database ID: 6
TABLE level scan performed.
- Pages Scanned................................: 9
- Extents Scanned..............................: 6
- Extent Switches..............................: 5
- Avg. Pages per Extent........................: 1.5
- Scan Density [Best Count:Actual Count].......: 33.33% [2:6]
- Logical Scan Fragmentation ..................: 0.00%
- Extent Scan Fragmentation ...................: 16.67%
- Avg. Bytes Free per Page.....................: 673.2
- Avg. Page Density (full).....................: 91.68%

By default, DBCC SHOWCONTIG scans the page chain at the leaf level of the specified index and keeps track of
the following values:
 Average number of bytes free on each page (Avg. Bytes Free per Page)
 Number of pages accessed (Pages scanned)
 Number of extents accessed (Extents scanned)
 Number of times a page had a lower page number than the previous page in the scan (This value for Out of
order pages is not displayed, but it is used for additional computations.)
 Number of times a page in the scan was on a different extent than the previous page in the scan (Extent
Switches)

SQL Server also keeps track of all the extents that have been accessed, and then it determines how many gaps are in
the used extents. An extent is identified by the page number of its first page. So, if extents 8, 16, 24, 32, and 40 make
up an index, there are no gaps. If the extents are 8, 16, 24, and 40, there is one gap. The value in DBCC
SHOWCONTIG's output called Extent Scan Fragmentation is computed by dividing the number of gaps by the
number of extents, so in this example the Extent Scan Fragmentation is ¼, or 25 percent. A table using extents 8,
24, 40, and 56 has 3 gaps, and its Extent Scan Fragmentation is ¾, or 75 percent. The maximum number of gaps
is the number of extents - 1, so Extent Scan Fragmentation can never be 100 percent.

The value in DBCC SHOWCONTIG's output called Logical Scan Fragmentation is computed by dividing the
number of Out Of Order Pages by the number of pages in the table. This value is meaningless in a heap.

You can use either the Extent Scan Fragmentation value or the Logical Scan Fragmentation value to determine
the general level of fragmentation in a table. The lower the value, the less fragmentation there is. Alternatively, you can
use the value called Scan Density, which is computed by dividing the optimum number of extent switches by the
actual number of extent switches. A high value means that there is little fragmentation. Scan Density is not valid if the
table spans multiple files, so all in all it is less useful than the other values.

You can use DBCC SHOWCONTIG to have the report returned in a format that can be easily inserted into a table.
If you have a table with the appropriate columns and datatypes, you can execute the following:
INSERT INTO CONTIG_TABLE
EXEC ('dbcc showcontig ([order details])
with all_indexes, tableresults')

Here's a subset of the results:

ObjectName IndexId Level Pages Rows
----------------- ----------- ----------- ----------- -----------
Order Details 1 0 9 2155
This document is created with the unregistered version of CHM2PDF Pilot

Order Details 2 0 4 2155

Order Details 3 0 4 2155

Many more columns are returned if you specify TABLERESULTS. In addition to the values that the nontabular
output returns, you also get the values listed in Table 8-3.

Table 8-3. Additional columns returned when TABLERESULTS is specified.

Column Meaning
ObjectName Name of the table or view processed.
ObjectId ID of the object processed.
IndexName Name of the index processed (NULL for a heap).
IndexId ID of the index (0 for a heap).
Level of the index. Level 0 is the leaf (or data) level of the
index. The level number increases moving up the tree
toward the index root. The level is 0 for a heap. By
Level
default, only level 0 is reported. If you specify
ALL_LEVELS along with TABLERESULTS, you get a
row for each level of the index.
Number of pages comprising that level of the index or the
Pages
entire heap.
Number of data or index records at that level of the index.
Rows For a heap, this is the number of data records in the entire
heap.
Minimum record size in that level of the index or the entire
MinimumRecordSize
heap.
Maximum record size in that level of the index or the
MaximumRecordSize
entire heap.
Average record size in that level of the index or the entire
AverageRecordSize
heap.
Number of forwarded records in that level of the index or
ForwardedRecords
the entire heap.
Number of extents in that level of the index or the entire
Extents
heap.

One last option available to DBCC SHOWCONTIG is the FAST option. The command takes less time to run when
you specify this option, as you might imagine, because it gathers only data that is available from the IAM pages and
the nonleaf levels of the index, and it returns only these values:
 Pages Scanned
 Extent Switches
 Scan Density
 Logical Scan Fragmentation

Since the level above the leaf has pointers to every page, SQL Server can determine all the page numbers and
determine the Logical Scan Fragmentation. In fact, it can also use the IAM pages to determine the Extent Scan
Fragmentation. However, the purpose of the FAST option is to determine whether a table would benefit from online
defragmenting, and since online defragmenting cannot change the Extent Scan Fragmentation, there is little benefit in
reporting it. I'll talk about online defragmenting in the next section.

Removing Fragmentation
This document is created with the unregistered version of CHM2PDF Pilot

Several methods are available for removing fragmentation from an index. First, you can rebuild the index and have
SQL Server allocate all new contiguous pages for you. You can do this by using a simple DROP INDEX and
CREATE INDEX, but I've already discussed some reasons why this is not optimal. In particular, if the index supports
a constraint, you can't use the DROP INDEX command. Alternatively, you can use DBCC DBREINDEX, which can
rebuild all the indexes on a table in a single operation, or you can use the drop_existing clause along with CREATE
INDEX.

The drawback of these methods is that the table is unavailable while the index is being rebuilt. If you are rebuilding
only nonclustered indexes, there is a shared lock on the table, which means that no modifications can be made, but
other processes can SELECT from the table. Of course, they cannot take advantage of the index you're rebuilding, so
the query might not perform as well as it should. If you're rebuilding the clustered index, SQL Server takes an
exclusive lock, and no access is allowed at all.

SQL Server 2000 allows you to defragment an index without completely rebuilding it. In this release, DBCC
INDEXDEFRAG reorders the leaf level pages into physical order as well as logical order, but using only the pages
that are already allocated to the leaf level. It basically does an in-place ordering, similar to a sorting technique called
bubble-sort. This can reduce the logical fragmentation to 0 to 2 percent, which means that an ordered scan through
the leaf level will be much faster. In addition, DBCC INDEXDEFRAG compacts the pages of an index, based on the
original fillfactor, which is stored in the sysindexes table. This doesn't mean that the pages always end up with the
original fillfactor, but SQL Server uses that as a goal. The process does try to leave at least enough space for one
average-size row after the defragmentation takes place. In addition, if a lock cannot be obtained on a page during the
compaction phase of DBCC INDEXDEFRAG, SQL Server skips the page and doesn't go back to it. Any empty
pages created as a result of this compaction are removed. Here's an example:
DBCC INDEXDEFRAG(0, 'charge', 1)

Here's the output:

Pages Scanned Pages Moved Pages Removed
------------- ----------- -------------
673 668 12

The algorithm SQL Server uses for DBCC INDEXDEFRAG finds the next physical page in a file belonging to the
leaf level and the next logical page in the leaf level with which to swap it. It finds the next physical page by scanning
the IAM pages for that index. The single-page allocations are not included in this release. Pages on different files are
handled separately. The algorithm finds the next logical page by scanning the leaf level of the index. After each page
move, all locks and latches are dropped and the key of the last page moved is saved. The next iteration of the
algorithm uses the key to find the next logical page. This allows other users to update the table and index while DBCC
INDEXDEFRAG is running.

For example, suppose the leaf level of an index consists of these pages, in this order:
47 22 83 32 12 90 64

The first step is to find the first physical page, which is 12, and the first logical page, which is 47. These pages are
then swapped, using a temporary buffer as a holding area. After the first swap, the leaf level looks like this:
12 22 83 32 47 90 64

The next physical page is 22, which is the same as the next logical page, so no work is done. The next physical page
is 32, which is swapped with the next logical page, 83, to look like this:
12 22 32 83 47 90 64
This document is created with the unregistered version of CHM2PDF Pilot

After the next swap of 47 with 83, the leaf level looks like this:
12 22 32 47 83 90 64

The 64 is swapped with 83:

12 22 32 47 64 90 83

And finally, 83 and 90 are swapped:

12 22 32 47 64 83 90

Keep in mind that DBCC INDEXDEFRAG uses only pages that are already part of the index leaf level. No new
pages are allocated. In addition, the defragmenting can take quite a while on a large table. You get a report every five
minutes on the estimated percentage completed.
This document is created with the unregistered version of CHM2PDF Pilot

34

Special Indexes
SQL Server 2000 allows you to create two special kinds of indexes: indexes on computed columns and indexes on
views. Without indexes, both of these constructs are purely logical. There is no physical storage for the data involved.
A computed column is not stored with the table data; it is recomputed every time a row is accessed. A view does not
save any data; it basically saves a SELECT statement that is reexecuted every time the data in the view is accessed.
With the new special indexes, SQL Server actually materializes what was only logical data into the physical leaf level
of an index.

Prerequisites
Before you can create indexes on either computed columns or views, certain prerequisites must be met. The biggest
issue is that SQL Server needs to be able to guarantee that given the identical base table data, the same values will
always be returned for any computed columns or for the rows in a view. To guarantee that the same values will
always be generated, these special indexes have certain requirements, which fall into three categories. First, a number
of session-level options must be set to a specific value. Second, there are some restrictions on the functions that can
be used within the column or view definition. The third requirement, which applies only to indexed views, is that the
tables that the view is based on must meet certain criteria.

SET Options
The following seven SET options can affect the result value of an expression or predicate, so you must set them as
shown to create indexed views or indexes on computed columns:
SET ARITHABORT ON
SET CONCAT_NULL_YIELDS_NULL ON
SET QUOTED_IDENTIFIER ON
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
SET NUMERIC_ROUNDABORT OFF

Note that all the options have to be ON except the NUMERIC_ROUNDABORT option, which has to be OFF. If
any of these options are not set as specified, you'll get an error message when you create a special index. In addition,
if you've already created one of these indexes and then you attempt to modify the column or view on which the index
is based, you'll get an error. If you issue a SELECT that normally should use the index, and if the seven SET options
do not have the values indicated, the index will be ignored but no error will be generated.

You can also use the property function SESSIONPROPERTY to test whether you have each option appropriately
set. A returned value of 1 means that the setting is ON, and a 0 means OFF. The following example checks the
current session setting for the option NUMERIC_ROUNDABORT:
SELECT SessionProperty('NUMERIC_ROUNDABORT')

Permissible Functions
A function is either deterministic or nondeterministic. If the function returns the same result every time it is called
with the same set of input values, it is deterministic. If it can return different results when called with the same set of
This document is created with the unregistered version of CHM2PDF Pilot

input values, it is nondeterministic. For the purposes of indexes, a function is considered deterministic if it always
returns the same values for the same input values when the seven SET options have the required settings. Any function
used in a computed column's definition or used in the SELECT list or WHERE clause of an indexable view must be
deterministic.

Table 8-4 lists the deterministic and nondeterministic functions in SQL Server 2000. Where it might not be obvious, I
have included an indication of why a particular function is nondeterministic—that is, why you might not always get the
same result when the function is called with the same input values.

Table 8-4. SQL Server 2000 deterministic and nondeterministic functions.

Function Deterministic or Nondeterministic Attribute
Aggregate functions (SUM, AVG, and so on)

(Note that you cannot use aggregate functions in Deterministic

computed columns—as I'll discuss below—but you can
use some of them in indexed views.)
Configuration Functions All are Nondeterministic
Cursor Functions All are Nondeterministic
Date and Time Functions:
DATEADD(datepart, number, date) Deterministic
DATEDIFF(datepart, startdate, enddate) Deterministic
Nondeterministic (the result is language dependent,
DATENAME particularly for the value returned when you're looking for
the names of months or days of the week.)
Deterministic (except for DATEPART(dw,date) because
DATEPART
the value depends on the current setting of DATEFIRST)
DAY (date) Deterministic
GETDATE Nondeterministic
GETUTCDATE Nondeterministic
MONTH (date) Deterministic
YEAR (date) Deterministic
Mathematical Functions:
RAND() Nondeterministic
The rest of mathematical functions, including
Deterministic
RAND(SEED)
Metadata functions (object info) All are Nondeterministic
Security functions (such as IS_MEMBER) All are Nondeterministic
All are Deterministic except CHARINDEX and
String functions (such as ASCII and STR)
PATINDEX
System Functions:
APP_NAME Nondeterministic
Nondeterministic if the datatype is datetime,
CAST
smalldatetime, or sql_variant. Deterministic otherwise.
Nondeterministic if the datatype is expression) datetime,
CONVERT (data_type[(length)],
smalldatetime, or sql_variant. Deterministic otherwise.
CONVERT (data_type[(length)], expression, style) Deterministic
COALESCE Deterministic
CURRENT_TIMESTAMP Nondeterministic
CURRENT_USER Nondeterministic
This document is created with the unregistered version of CHM2PDF Pilot

DATALENGTH Deterministic
@@ERROR Nondeterministic
FORMATMESSAGE Nondeterministic
GETANSINULL Nondeterministic
HOST_ID Nondeterministic
HOST_NAME Nondeterministic
IDENT_INCR Nondeterministic
IDENT_SEED Nondeterministic
@@IDENTITY Nondeterministic
IDENTITY (function) Nondeterministic
Nondeterministic (because it depends on
DATEFORMAT)

ISDATE (expression) (Note that this function can be deterministic if used with
the CONVERT function, with CONVERT style
parameter specified, and style not equal to 0, 100, 9, or
109.)
ISNULL Deterministic
ISNUMERIC Deterministic
NEWID Nondeterministic
NULLIF Deterministic
PARSENAME Deterministic
PERMISSIONS Nondeterministic
@@ROWCOUNT Nondeterministic
SESSION_USER Nondeterministic
STATS_DATE Nondeterministic
SYSTEM_USER Nondeterministic
@@TRANCOUNT Nondeterministic
USER_NAME Nondeterministic
CHECKSUM(select list) Deterministic except for CHECKSUM(*)
System statistical functions such as @@cpu_busy,
Nondeterministic
@@idle
Text and Image Functions:
PATINDEX Nondeterministic
TEXTPTR Nondeterministic
TEXTVALID Nondeterministic
OPERATORS (such as +, -, /, *, string +, - neg) Deterministic
Implicit Date Conversion Functions:
Nondeterministic (because it depends on
String -> date
DATEFORMAT)

This might seem like quite a restrictive list, but the same restrictions apply to functions you use in your own
user-defined functions (UDFs)—that is, your own functions cannot be based on any nondeterministic built-in function.
I'll talk about UDFs in Chapter 11.

Schema Binding
This document is created with the unregistered version of CHM2PDF Pilot

To create an indexed view, a requirement on the table itself is that you don't want the definition of any underlying
object's schema to change. To prevent a change in schema definition, SQL Server 2000's CREATE VIEW statement
allows the WITH SCHEMABINDING option. When you specify WITH SCHEMABINDING, the SELECT
statement that defines the view must include the two-part names (owner.object) of all referenced tables. You can't
drop or alter tables participating in a view created with the SCHEMABINDING clause unless you've dropped that
view or changed the view so that it no longer has schema binding. Otherwise, SQL Server raises an error. If any of
the tables on which the view is based is owned by someone other than the user creating the view, the view creator
doesn't automatically have the right to create the view with schema binding because that would restrict the table's
owner from making changes to her own table. The table owner must grant another user REFERENCES permission in
order for that user to create a view with schema binding on that table. We'll see an example of schema binding in a
moment.

Indexes on Computed Columns

SQL Server 2000 allows you to build indexes on deterministic computed columns where the resulting data type is
otherwise indexable. This means that the column's datatype cannot be text, ntext, or image. Such a computed
column can participate at any position of an index or in a PRIMARY KEY or UNIQUE constraint. You cannot
define a FOREIGN KEY, CHECK, or DEFAULT constraint on a computed column, and computed columns are
always considered nullable, unless you enclose the expression in the ISNULL function.. You cannot create indexes on
any computed columns in system tables. When you create an index on computed columns, the seven SET options
must first have the correct values set.

Here's an example:
CREATE TABLE t1 (a int, b as 2*a)
GO
CREATE INDEX i1 on t1 (b)
GO

If any of your seven SET options don't have the correct values when you create the table, you get this message when
you try to create the index:
Server: Msg 1934, Level 16, State 1, Line 2
CREATE INDEX failed because the following SET options have
incorrect settings: '<OPTION NAME>'.

If more than one option has an incorrect value, the error message reports only one incorrectly set option.

Here's an example that creates a table with a nondeterministic computed column:

CREATE TABLE t2 (a int, b datetime, c AS datename(mm, b))
GO
CREATE INDEX i2 on t2 (c)
GO

When you try to create the index on the computed column c, you get this error:
Server: Msg 1933, Level 16, State 1, Line 1
Cannot create index because the key column 'c' is
non-deterministic or imprecise.

Column c is nondeterministic because the month value of datename can have different values depending on the
language you're using.
This document is created with the unregistered version of CHM2PDF Pilot

Using the COLUMNPROPERTY Function

You can use the new IsDeterministic property to determine before you create an index on a computed column
whether that column is deterministic. If you specify this property, the COLUMNPROPERTY function returns 1 if the
column is deterministic and 0 otherwise. The result is undefined for noncomputed columns, so you should consider
checking the IsComputed property before you check the IsDeterministic property. The following example detects that
column c in table t2 in the previous example is nondeterministic:
SELECT COLUMNPROPERTY(object_id('t2'), 'c', 'IsDeterministic')

The value 0 is returned, which means that column c is nondeterministic. Note that the COLUMNPROPERTY
function requires an object ID for the first argument and a column name for the second argument.

Implementation of a Computed Column

If you create a clustered index on a computed column, the computed column is no longer a virtual column in the table.
Its computed value will physically exist in the rows of the table, which is the leaf level of the clustered index. Updates
to the columns that the computed column is based on will update the computed column in the table itself. For
example, in the t1 table created previously, if we insert a row with the value 10 in column a, the row will be created
with both the values 10 and 20 in the actual data row. If we then update the 10 to 15, the second column will be
automatically updated to 30.

In Chapter 16, I'll revisit indexes on computed columns and show you some situations where you can make good use
of them.

Indexed Views
Indexed views in SQL Server 2000 are similar to what other products call materialized views. The first index you
must build on a view is a clustered index, and since the clustered index contains all the data at its leaf level, this index
actually does materialize the view. The view's data is physically stored at the leaf level of the clustered index.

Additional Requirements
In addition to the requirement that all functions used in the view be deterministic and that the seven SET options be
set to the appropriate values, the view definition can't contain any of the following:
 TOP
 text, ntext, or image columns
 DISTINCT
 MIN, MAX, COUNT(*), COUNT(<expression>), STDEV, VARIANCE, AVG
 SUM on a nullable expression
 A derived table
 The ROWSET function
 Another view (you can reference only base tables)
 UNION
 Subqueries, OUTER joins, or self-joins
 Full-text predicates (CONTAINS, FREETEXT)
 COMPUTE, COMPUTE BY
 ORDER BY

Also, if the view definition contains GROUP BY, you must include the aggregate COUNT_BIG(*) in the SELECT
list. COUNT_BIG returns a value of the datatype BIGINT, which is an 8-byte integer. A view that contains GROUP
This document is created with the unregistered version of CHM2PDF Pilot

BY can't contain HAVING, CUBE, ROLLUP, or GROUP BY ALL. Also, all GROUP BY columns must appear in
the SELECT list. Note that if your view contains both SUM and COUNT_BIG(*), you can compute the equivalent
of the AVG function even though AVG is not allowed in indexed views. Although these restrictions might seem
severe, remember that they apply to the view definitions, not to the queries that might use the indexed views.

To verify that you've met all the requirements, you can use the OBJECTPROPERTY function's IsIndexable
property. The following query tells you whether you can build an index on a view called Product Totals:
SELECT ObjectProperty(object_id('Product_Totals'), 'IsIndexable')

A return value of 1 means you've met all requirements and can build an index on the view.

Creating an Indexed View

The first step to building an index on a view is to create the view itself. Here's an example from the Northwind
database:
USE northwind
GO
CREATE VIEW Product_Totals
WITH SCHEMABINDING
AS
select productid, total_volume = sum(unitPrice * Quantity),
total_qty = sum(Quantity) , number = count_big(*)
from dbo."order details"
group by productid

Note the WITH SCHEMABINDING clause and the specification of the owner name (dbo) for the table. At this
point, we have a normal view—a stored SELECT statement that uses no storage space. In fact, if we run the system
stored procedure sp_spaceused on this view, we'll get this error message:
Server: Msg 15235, Level 16, State 1, Procedure sp_spaceused, Line 91
Views do not have space allocated.

To create an indexed view, you need to create an index. The first index you create on a view must be a unique
clustered index. Clustered indexes are the only type of SQL Server index that contains data; the clustered index on a
view contains all the data that makes up the view definition. This statement defines a unique clustered index, PV_IDX,
for the view:
CREATE UNIQUE CLUSTERED INDEX PV_IDX on Product_Totals(productid)

After you create the index, you can rerun sp_spaceused. My output looks like this:
Name rows reserved data index_size unused
----------------- ------- ---------- ----- ---------- ------
Product_Totals 77 24 KB 8 KB 16 KB 0 KB

Data that comprises the indexed view is persistent, with the indexed view storing the data in the clustered index's leaf
level. You could construct something similar by using temporary tables to store the data you're interested in. But a
temporary table is static and doesn't reflect changes to underlying data. In contrast, SQL Server automatically
maintains indexed views, updating information stored in the clustered index whenever anyone changes data that affects
the view.
This document is created with the unregistered version of CHM2PDF Pilot

After you create the unique clustered index, you can create multiple nonclustered indexes on the view. You can
determine whether a view is indexed by using the OBJECTPROPERTY function's IsIndexed property. For the
Total_Products indexed view, the following statement returns a 1, which means that the view is indexed:
SELECT ObjectProperty(object_id('Product_Totals'), 'IsIndexed')

Executing the system stored procedure sp_help or sp_helpindex returns complete information about any indexes on
the view.

Using an Indexed View

One of the most valuable benefits of indexed views is that your queries don't have to directly reference a view to use
the index on the view. Consider the Product_Totals indexed view. Suppose you issue the following SELECT
statement:
SELECT productid, total_qty = sum(Quantity)
FROM dbo."order details"
GROUP BY productid

SQL Server's query optimizer will realize that the precomputed sums of all the Quantity values for each productid
are already available in the index for the Product_Totals view. The query optimizer will evaluate the cost of using that
indexed view in processing the query. But just because you have an indexed view doesn't mean that the query
optimizer will always choose it for the query's execution plan. In fact, even if you reference the indexed view directly
in the FROM clause, the query optimizer might decide to directly access the base table instead. In Chapters 15 and
16, I'll tell you more about how the query optimizer decides whether to use indexed views and how you can tell if
indexed views are being used for a query.
This document is created with the unregistered version of CHM2PDF Pilot

34

Using an Index
This chapter has primarily been concerned with the structure of indexes and the nuances of creating and managing
them. Hopefully, you're aware of at least some of the places that indexes can be useful in your SQL Server
applications. I'll list a few of the situations in which you can benefit greatly from indexes; I'll also revisit most of these
situations in Chapters 15 and 16 when I discuss SQL Server query processing and tuning in detail.

Looking for Rows

The most straightforward use of an index is to help SQL Server find one or more rows in a table that satisfy a certain
condition. For example, if you're looking for all the customers who live in a particular state (for example, your
WHERE clause is WHERE state = 'WI'), you can use an index on the state column to find those rows more quickly.
SQL Server can traverse the index from the root, comparing index key values to 'WI', to determine whether 'WI'
exists in the table and then find the data associated with that value.

Joining
A typical join tries to find all the rows in one table that match rows in anther table. Of course, you can have joins that
aren't looking for exact matching, but you're looking for some sort of relationship between tables, and equality is by
far the most common relationship. A query plan for a join frequently starts with one of the tables, finds the rows that
match the search conditions, and then uses the join key in the qualifying rows to find matches in the other table. An
index on the join column in the second table can be used to quickly find the rows that match.

Sorting
A clustered index stores the data logically in sorted order. The data pages are linked together in order of the
clustering keys. If you have a query to ORDER BY the clustered keys or by the first column of a composite clustered
key, SQL Server does not have to perform a sort operation to return the data in sorted order.

If you have a nonclustered index on the column you're sorting by, the sort keys themselves are in order at the leaf
level. For example, consider the following query:
SELECT firstname, lastname, state, zipcode
FROM customers
ORDER BY zipcode

The nonclustered index will have all the zip codes in sorted order. However, we want the data pages themselves to
be accessed to find the firstname and lastname values associated with the zipcode. The query optimizer will decide
if it's faster to traverse the nonclustered index leaf level and from there access each data row or to just perform a sort
on the data. If you're more concerned with getting the first few rows of data as soon as possible and less concerned
with the overall time to return all the results, you can force SQL Server to use the nonclustered index with the FAST
hint. I'll tell you all about hints in Chapter 16.

Inverse Indexes
SQL Server allows you to sort in either ascending or descending order, and if you have a clustered index on the sort
This document is created with the unregistered version of CHM2PDF Pilot

column, SQL Server can use that index and avoid the actual sorting. The pages at the leaf level of the clustered index
are doubly linked, so SQL Server can use a clustered index on lastname to satisfy this query
SELECT * FROM customers ORDER BY lastname DESC

as easily as it can use the same index for this query:

SELECT * FROM customers ORDER BY lastname

Earlier in the chapter, I mentioned that SQL Server 2000 allows you to create descending indexes. Why would you
need them if an ascending index can be scanned in reverse order? The real benefit is when you want to sort with a
combination of ascending and descending order. For example, take a look at this query:
SELECT * FROM employees
ORDER BY lastname, salary DESC

The default sort order is ascending, so the query wants the names ordered alphabetically by last name, and within
duplicate last names, the rows should be sorted with the highest salary first and lowest salary last. The only kind of
index that can help SQL Server avoid actually sorting the data is one with the lastname and salary columns sorted in
opposite orders, such as this one:
CREATE CLUSTERED INDEX name_salary_indx
ON employees (lastname, salary DESC)

If you execute sp_help or sp_helpindex on a table, SQL Server will indicate whether the index is descending by
placing a minus (-) after the index key. Here's a subset of some sp_helpindex output:
index_name index_description index_keys
--------------------- -------------------------------- -------------------
lastname_salary_indx nonclustered located on PRIMARY LastName, Salary(-)

Grouping
One way that SQL Server can perform a GROUP BY operation is by first sorting the data by the grouping column.
For example, if you want to find out how many customers live in each state, you can write a query with a GROUP
BY state clause. A clustered index on state will have all the rows with the same value for state in logical sequence, so
the sorting and grouping operations will be very fast.

Maintaining Uniqueness
Creating a unique index (or defining a PRIMARY KEY or UNIQUE constraint that builds a unique index) is by far
the most efficient method of guaranteeing that no duplicate values are entered into a column. By traversing an index
tree to determine where a new row should go, SQL Server can detect within a few page reads that a row already has
that value. Unlike all the other uses of indexes described in this section, using unique indexes to maintain uniqueness
isn't just one option among others. Although SQL Server might not always traverse a particular unique index to
determine where to try to insert a new row, SQL Server will always use the existence of a unique index to verify
whether a new set of data is acceptable.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
In this chapter, I showed you how SQL Server indexes organize the data on disk and help you access your data
more quickly than if no indexes existed. Indexes are organized as B-trees, which means that you will always traverse
through the same number of index levels when you traverse from the root to any leaf page. To use an index to find a
single row of data, SQL Server never has to read more pages than there are levels in an appropriate index.

You also learned about all the options available when you create an index, how to determine the amount of space an
index takes up, and how to predict the size of an index that doesn't have any data yet.

Indexes can become fragmented in SQL Server 2000, but the performance penalty for fragmentation is usually much
less than in earlier versions of SQL Server. When you want to defragment your indexes, you have several methods to
choose from, one of which allows the index to continue to be used by other operations, even while the
defragmentation operation is going on.

Finally, you learned about some of the situations in which you can get the greatest performance gains by having the
appropriate indexes on your tables.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 9

Modifying Data
In Chapter 7, we looked at how to query data. In this chapter, we'll look at inserting, updating, and deleting data in
tables. We'll cover the basic SQL commands for these operations but focus on Microsoft SQL Server-specific
behaviors. This chapter considers each statement as an independent, atomic operation. Chapter 12 discusses
combining multiple statements into an all-or-nothing atomic operation (that is, a transaction). The use of constraints
(covered in Chapter 6) is closely related to the topic of modifying data, but this chapter will assume that the
modifications being performed satisfy any constraints on a table.
This document is created with the unregistered version of CHM2PDF Pilot

34

Basic Modification Operations

SQL has three basic data modification statements: INSERT, UPDATE, and DELETE. I used these in some
examples in previous chapters, assuming that you were already familiar with them. Here, I'll quickly review the most
typical operations. Along with SELECT, the INSERT, UPDATE, and DELETE statements are referred to as DML,
or data manipulation language. (DML is sometimes mistakenly referred to as data modification language, in
which case SELECT couldn't be included, because it doesn't modify anything.) SELECT can manipulate data as the
data is returned by using functions, aggregates, grouping, or the like. (Creation operations such as CREATE TABLE
are DDL—data definition language—whereas security operations such as GRANT/DENY/REVOKE are DCL—
data control language.)

INSERT
You generally use INSERT to add one row to a table. Here's the most common form of INSERT:
INSERT [INTO] {table_name|view_name} [(column_list)]
VALUES value_list

In SQL Server, the use of INTO is always optional, but ANSI SQL specifies using INTO. If you're providing a
value for every column in the table and the values appear in the exact order in which the columns were defined, the
column list is optional. The one exception to this is when the table has an identity column, which you must not include
in the list of values. Later in this chapter, you'll learn more about inserting rows into a table that contains an identity
column. If you omit a value for one or more columns or if the order of your values differs from the order in which the
columns were defined for the table, you must use a named columns list. If you don't provide a value for a particular
column, that column must allow NULL or must have a default declared for it. You can use the keyword DEFAULT
as a placeholder. You can explicitly enter NULL for a column, or SQL Server will implicitly enter NULL for an
omitted column for which no default value exists.

You can issue an INSERT statement for a view, but the row is always added to only one underlying table. If the view
is an indexed view, all indexes will be updated to reflect the new data. You can insert data into a view as long as
values (or defaults) are provided for all the columns of the underlying table into which the new row is being added.
Partitioned views have a few additional restrictions, some of which I discussed in Chapter 6. I'll tell you more about
partitioned views when I discuss query tuning in Chapter 16.

Following are some simple examples of using INSERT statements in a table that's similar to publishers in the pubs
sample database. The CREATE TABLE statement is shown so that you can easily see column order and you can see
which columns allow NULL and which have defaults declared.
CREATE TABLE publishers2
(
pub_id int NOT NULL PRIMARY KEY IDENTITY,
pub_name varchar(40) NULL DEFAULT ('Anonymous'),
city varchar(20) NULL,
state char(2) NULL,
country varchar(30) NOT NULL DEFAULT('USA')
)
GO

INSERT publishers2 VALUES ('AAA Publishing', 'Vancouver', 'BC',

'Canada')
This document is created with the unregistered version of CHM2PDF Pilot

INSERT INTO publishers2 VALUES ('Best Publishing', 'Mexico City',

NULL, 'Mexico')

INSERT INTO publishers2 (pub_name, city, state, country)

VALUES ('Complete Publishing', 'Washington', 'DC', 'United States')

INSERT publishers2 (state, city) VALUES ('WA', 'Poulsbo')

INSERT publishers2 VALUES (NULL, NULL, NULL, DEFAULT)

INSERT publishers2 VALUES (DEFAULT, NULL, 'WA', DEFAULT)

INSERT publishers2 VALUES (NULL, DEFAULT, DEFAULT, DEFAULT)

INSERT publishers2 DEFAULT VALUES

GO

The table has these values:

pub_id pub_name city state country
------ ------------------- ----------- ----- -------------
0001 AAA Publishing Vancouver BC Canada
0002 Best Publishing Mexico City NULL Mexico
0003 Complete Publishing Washington DC United States
0004 Anonymous Poulsbo WA USA
0005 NULL NULL NULL USA
0006 Anonymous NULL WA USA
0007 NULL NULL NULL USA
0008 Anonymous NULL NULL USA

These INSERT examples are pretty self-explanatory, but you should be careful of the following:
 If a column is declared to allow NULL and it also has a default bound to it, omitting the column entirely from
the INSERT statement results in the default value being inserted, not the NULL value. This is also true if the
column is declared NOT NULL.
 Other than for quick-and-dirty, one-time use, you're better off providing the column list to explicitly name the
columns. Your INSERT statement will then still work even if you add a column via ALTER TABLE.
 If one of the INSERT statements in this example had failed, the others would have continued and succeeded.
Even if I had wrapped multiple INSERT statements in a BEGIN TRAN/COMMIT TRAN block (and did
nothing more than that), a failure of one INSERT (because of an error such as a constraint violation or
duplicate value) wouldn't cause the others to fail. This behavior is expected and proper. If you want all the
statements to fail when one statement fails, you must add error handling. (Chapter 12 covers error handling in
more detail.)
 The special form of the INSERT statement that uses DEFAULT VALUES and no column list is a shorthand
method that enables you to avoid supplying the keyword DEFAULT for each column.
 You can't specify the column name for a column that has the IDENTITY property, and you can't use the
DEFAULT placeholder for a column with the IDENTITY property. (This would be nice, but SQL Server
doesn't currently work this way.) You must completely omit a reference to the identity column. To explicitly
provide a value for the column, you must use SET IDENTITY_INSERT ON. You can use the DEFAULT
VALUES clause, however.

DEFAULT and NULL Precedence

You should understand the general behavior of INSERT in relation to NULL and DEFAULT precedence. If you
omit a column from the column list and the values list, the column takes on the default value, if one exists. If a default
value doesn't exist, SQL Server tries a NULL value. An error results if the column has been declared NOT NULL. If
NULL is explicitly specified in the values list, a column is set to NULL (assuming it allows NULL) even if a default
exists. When you use the DEFAULT placeholder on a column allowing NULL and no default has been declared,
This document is created with the unregistered version of CHM2PDF Pilot

NULL is inserted for that column. An error results in a column declared NOT NULL without a default if you specify
NULL or DEFAULT or if you omit the value entirely.

Table 9-1 summarizes the results of an INSERT statement that omits columns, specifies NULL, or specifies
DEFAULT, depending on whether the column is declared NULL or NOT NULL and whether it has a default
declared.

Table 9-1. The effects of an INSERT statement that omits columns, specifies NULL, or specifies DEFAULT.

Expressions in the VALUES Clause

So far, the INSERT examples have used only constant values in the VALUES clause of INSERT. In fact, you can
use a scalar expression such as a function or a local variable in the VALUES clause. (I'll show you functions and
variables in Chapter 10; you can take a quick look ahead if you need to.) You can't use an entire SELECT statement
as a scalar value even if you're certain that it returns only one row and one column. However, you can take the result
of that SELECT statement and assign it to a variable, and then you can use that variable in the VALUES clause. In
the next section, you'll see how a SELECT statement can completely replace the VALUES clause.

Here's a contrived example that demonstrates how functions, expressions, arithmetic operations, string concatenation,
and local variables are used in the VALUES clause of an INSERT statement:
CREATE TABLE mytable
(
int_val int,
smallint_val smallint,
numeric_val numeric(8, 2),
tiny_const tinyint,
float_val float,
date_val datetime,
char_strng char(10)
)
GO

DECLARE @myvar1 numeric(8, 2)

SELECT @myvar1=65.45

INSERT mytable (int_val, smallint_val, numeric_val, tiny_const,

float_val, date_val, char_strng)
VALUES
(OBJECT_ID('mytable'), @@spid, @myvar1 / 10.0, 5,
This document is created with the unregistered version of CHM2PDF Pilot

SQRT(144), GETDATE(), REPLICATE('A', 3) + REPLICATE('B', 3))

GO

SELECT * FROM mytable

GO

The results will look something like this:

int_val smallint_val Numeric_val tiny_const float_val
---------- ------------ ----------- ---------- ---------
2082106458 56 6.55 5 12.0

date_val char_string
----------------------- -------------------
2000-06-27 12:22:55.040 AAABBB

If you run this query, your values will be slightly different because your table will have a different object_id value,
your process ID (spid) might be different, and you'll be executing this query at a different date and time.

Multiple-Row INSERT Statements

The most typical use of INSERT is to add one row to a table. However, two special forms of INSERT
(INSERT/SELECT and INSERT/EXEC) and a special SELECT statement (SELECT INTO) allow you to add
multiple rows of data at once. Note that you can use the system function @@ROWCOUNT after all these
statements to find out the number of rows affected.

INSERT/SELECT As mentioned earlier, you can use a SELECT statement instead of a VALUES clause with
INSERT. You get the full power of SELECT, including joins, subqueries, UNION, and all the other goodies. The
table you're inserting into must already exist; it can be a permanent table or a temporary table. The operation is
atomic, so a failure of one row, such as from a constraint violation, causes all rows chosen by the SELECT statement
to be thrown away. An exception to this occurs if a duplicate key is found on a unique index created with the
IGNORE_DUP_KEY option. In this case, the duplicate row is thrown out but the entire statement continues and isn't
aborted. You can, of course, also use expressions in the SELECT statement, and using the CONVERT function is
common if the target table has a datatype that's different from that of the source table.

For example, suppose I want to copy the authors table (in the pubs sample database) to a temporary table. But
rather than show the author ID as a char field in Social Security-number format with hyphens, I want to strip out
those hyphens and store the ID as an int. I want to use a single name field with the concatenation of the last and first
name. I also want to record the current date but strip off the time so that the internal time is considered midnight. If
you're working only with dates and want to ignore the time portion of datetime values, this is a good idea because it
avoids issues that occur when the time portions of columns aren't equal. I'll also record each author's area code—the
first three digits of the phone number. I need the author's state, but if it's NULL, I'll use WA instead.
CREATE TABLE #authors
(
au_id int PRIMARY KEY,
au_fullname varchar(60) NOT NULL,
date_entered smalldatetime NOT NULL,
area_code char(3) NOT NULL,
state char(2) NOT NULL
)
GO

INSERT INTO #authors

SELECT
CONVERT(int, SUBSTRING(au_id, 1, 3) + SUBSTRING(au_id, 5, 2)
This document is created with the unregistered version of CHM2PDF Pilot

+ SUBSTRING(au_id, 8, 4)),
au_lname + ', ' + au_fname,
CONVERT(varchar, GETDATE(), 102),
CONVERT(char(3), phone),
ISNULL(state, 'WA')
FROM authors

SELECT * FROM #authors

Here's the result:

au_id au_fullname date_entered area_code state
--------- ----------------- ------------------- --------- -----
172321176 White, Johnson 1998-08-19 00:00:00 408 CA
213468915 Green, Marjorie 1998-08-19 00:00:00 415 CA
238957766 Carson, Cheryl 1998-08-19 00:00:00 415 CA
267412394 O'Leary, Michael 1998-08-19 00:00:00 408 CA
274809391 Straight, Dean 1998-08-19 00:00:00 415 CA
341221782 Smith, Meander 1998-08-19 00:00:00 913 KS
409567008 Bennet, Abraham 1998-08-19 00:00:00 415 CA
427172319 Dull, Ann 1998-08-19 00:00:00 415 CA
472272349 Gringlesby, Burt 1998-08-19 00:00:00 707 CA
486291786 Locksley, Charlene 1998-08-19 00:00:00 415 CA
527723246 Greene, Morningstar 1998-08-19 00:00:00 615 TN
648921872 Blotchet-Halls, 1998-08-19 00:00:00 503 OR
Reginald
672713249 Yokomoto, Akiko 1998-08-19 00:00:00 415 CA
712451867 del Castillo, Innes 1998-08-19 00:00:00 615 MI
722515454 DeFrance, Michel 1998-08-19 00:00:00 219 IN
724089931 Stringer, Dirk 1998-08-19 00:00:00 415 CA
724809391 MacFeather, Stearns 1998-08-19 00:00:00 415 CA
756307391 Karsen, Livia 1998-08-19 00:00:00 415 CA
807916654 Panteley, Sylvia 1998-08-19 00:00:00 301 MD
846927186 Hunter, Sheryl 1998-08-19 00:00:00 415 CA
893721158 McBadden, Heather 1998-08-19 00:00:00 707 CA
899462035 Ringer, Anne 1998-08-19 00:00:00 801 UT
998723567 Ringer, Albert 1998-08-19 00:00:00 801 UT

I find that using the INSERT/SELECT is typically the most convenient way to quickly get some sample data into a
table because the SELECT statement can simply select constants. For example, to create a table with an int and a
varchar(10) column and then insert two rows into the table, I can use the following code:
CREATE TABLE demo (col1 int, col2 varchar(10))
GO

INSERT INTO demo SELECT 1, 'hello'

INSERT INTO demo SELECT 10, 'good-bye'
GO

INSERT/EXEC You can use INSERT with the results of a stored procedure or a dynamic EXECUTE statement
taking the place of the VALUES clause. This procedure is similar to INSERT/SELECT except that EXEC is used
instead. EXEC should return exactly one result set with types that match the table you've set up for it. You can pass
parameters if you're executing a stored procedure, use EXEC('string'), or even call out to extended procedures (your
own custom DLLs) or to remote procedures on other servers. By calling to a remote procedure on another server,
putting the data into a temporary table, and then joining on it, you get the capabilities of a distributed join.

For example, suppose I want to store the results of executing the sp_configure stored procedure in a temporary
table. (You can also do this with a permanent table.)
This document is created with the unregistered version of CHM2PDF Pilot

CREATE TABLE #config_out

(
name_col varchar(50),
minval int,
maxval int,
configval int,
runval int
)

INSERT #config_out
EXEC sp_configure

SELECT * FROM #config_out

Here's the result:

name_col minval maxval configval runval
-------------------- ------ ---------- --------- --------
allow updates 0 1 0 0
default language 0 9999 0 0
max text repl size (B) 0 2147483647 65536 65536
nested triggers 0 1 1 1
remote access 0 1 1 1
remote login timeout 0 2147483647 20 20
remote proc trans 0 1 0 0
remote query timeout 0 2147483647 600 600
show advanced options 0 1 0 0
user options 0 4095 0 0

If I want to execute the procedure against the remote server named dogfood, that's almost as easy (assuming that the
same table #config_out exists):
INSERT #config_out
EXEC dogfood.master.dbo.sp_configure

NOTE
Before executing a procedure on another server, you must perform several administrative steps to enable the servers
to communicate with each other. See the SQL Server documentation for details on linked servers. After you carry out
these simple steps, you can execute procedures on the remote server by simply including the name of the server as
part of the procedure name, as shown in the previous example.

SELECT INTO SELECT INTO is in many ways similar to INSERT/SELECT, but it directly builds the table rather
than requiring that the table already exist. In addition, SELECT INTO can operate as a minimally logged operation,
depending on your recovery model. When you're running in SIMPLE or BULK_LOGGED recovery mode, the
SELECT INTO operation can be a lot faster than an INSERT that adds the same data. (Chapter 5 discussed
recovery models.)

You can also use SELECT INTO with temporary tables (which are prefixed with # or ##). A user executing
SELECT INTO—whether to create a temporary or a permanent table—must have permission to select from the
source table and must also have CREATE TABLE permission because the SELECT INTO statement does both
actions.

SELECT INTO is handy. It's commonly used to copy a table, modify the datatype of a column, or switch the order
of columns. To illustrate the SELECT INTO statement, here's an equivalent operation to the earlier
INSERT/SELECT example. The results, including the column names of the new temporary table, appear identical to
those produced using INSERT/SELECT.
This document is created with the unregistered version of CHM2PDF Pilot

SELECT
CONVERT(int, SUBSTRING(au_id, 1, 3) + substring(au_id, 5, 2)
+ substring(au_id, 8, 4)) AS au_id,
au_lname + ', ' + au_fname AS au_fullname,
convert(varchar, getdate(), 102) AS date_entered,
convert(char(3), phone) AS area_code,
isnull(state, 'WA') AS state

INTO #authors
FROM authors

However, there's one important difference between the two procedures: the datatypes of the table created
automatically are slightly different from the datatypes declared explicitly using INSERT/SELECT. If the exact
datatypes are important, you can use CONVERT to cast them to the types you want in all cases or you can create
the table separately and then use INSERT/SELECT. However, if you want to specify a particular nullability value,
there is no way to do that. If a column is copied directly from the source to the destination, it keeps the same NULL
or NOT NULL property; if any conversion is done on the column, the new column will have the default nullability of
the connection. To force a nullability different from these defaults, you have to create the table before you copy the
data into it and specify NULL or NOT NULL for each column.

Another feature of SELECT INTO is that you can add an identity column to the new table by using the function
IDENTITY in the SELECT clause. The function takes up to three parameters. The first is the datatype, the second is
the seed, and the third is the increment. SQL Server automatically populates the rows with the appropriate identity
values if the table already has data in it. The following script creates a copy of the sales table in the pubs database,
adding an additional identity column to be the order ID for each row:
SELECT IDENTITY(int, 1, 1) AS OrderID, *
INTO NewSales
FROM Sales

BULK INSERT
SQL Server 2000 provides the BULK INSERT command for loading a flat file of data into a SQL Server table.
However, it offers no corresponding command for copying data out to a file. The file to be copied in must be either
local or available via a UNC name. Here's the general syntax for BULK INSERT:

BULK INSERT [['database_name'.]['owner'].]{'table_name' FROM data_file}

[WITH
(
[ BATCHSIZE [ = batch_size]]
[[,] CODEPAGE [ = ACP | OEM | RAW | code_page]]
[[,] CHECK_CONSTRAINTS]
[[,] DATAFILETYPE [ =
{'char' | 'native' | 'widechar' | 'widenative'}]]
[[,] FIELDTERMINATOR [ = 'field_terminator']]
[[,] FIRETRIGGERS]
[[,] FIRSTROW [ = first_row]]
[[,] FORMATFILE [ = 'format_file_path']]
[[,] KEEPIDENTITY]
[[,] KILOBYTES_PER_BATCH [ = kilobytes_per_batch]]
[[,] KEEPNULLS]
[[,] LASTROW [ = last_row]]
[[,] MAXERRORS [ = max_errors]]
[[,] ORDER ({column [ASC | DESC]} [, ...n])]
[[,] ROWTERMINATOR [ = 'row_terminator']]
[[,] TABLOCK]
)
This document is created with the unregistered version of CHM2PDF Pilot

To use the BULK INSERT command, the table must already exist. The number and datatypes of the columns must
be compatible with the data in the file that you plan to copy in.

Here's a simple example to give you a feel for what this command can do:

Create a simple table in the pubs database using the following batch:
USE pubs
CREATE TABLE mybcp
(
col1 char(3),
col2 INT
)

Use your favorite editor (such as Notepad) to save a text file named mydata.txt to the root directory of drive C.
Include the following text:
abc,1;def,2;ghi,3;

Load the file into the table using the following Transact-SQL statement:
BULK INSERT pubs.dbo.mybcp
FROM 'c:\mydata.txt'
WITH
(DATAFILETYPE = 'char',
FIELDTERMINATOR = ',', ROWTERMINATOR = ';')

NOTE
The data file must specify a valid path from the machine on which the SQL Server is running. If the file is a remote
file, you must specify a UNC name.

As the syntax specification shows, the BULK INSERT command has quite a few possible arguments. These
arguments are described in the online documentation, but because some of their descriptions might be unclear, I'll give
you the highlights in Table 9-2.

Table 9-2. Some of the BULK INSERT command arguments.

Argument Description
Each batch is copied to the server as a single transaction.
If an entire batch is successfully loaded, the data is
immediately written to disk. If a server failure occurs in
BATCHSIZE the middle of a batch, all rows in that batch will be lost but
all rows up to the end of the previous batch will still be
available. By default, all data in the specified data file is
one batch.
The data file is already sorted by the same columns in
column_list as the destination table. A clustered index on
the same columns, in the same order as the columns in
ORDER (column_list) where column_list = {column
column_list, must already exist in the table. Using a
[ASC | DESC] [,…n]}
sorted data file can improve the performance of the
BULK INSERT command. By default, BULK INSERT
assumes that the data file is unordered.
This document is created with the unregistered version of CHM2PDF Pilot

A table-level lock is acquired for the duration of the bulk

copy. It improves performance because of reduced lock
contention on the table. This special kind of BulkUpdate
lock is primarily compatible only with other BulkUpdate
locks, so a table can be loaded quickly using multiple
TABLOCK
clients running BULK INSERT in parallel. By default, no
table-level lock is acquired. You can still run parallel
BULK INSERTs, but their performance will be much
slower, similar to running individual INSERT statements. (
Chapter 14 covers locking in detail.)
Any constraints on the table are applied during the bulk
copy. By default, constraints aren't enforced, except for
PRIMARY KEY and UNIQUE constraints. Applying
constraints can have several benefits and is the
CHECK_CONSTRAINTS recommended approach. By checking the constraints on
the data, you know that the data you load will be valid. In
addition, SQL Server might be able to better optimize the
entire operation because it has advance knowledge of the
kind of data that will be loaded.
Any INSERT triggers on the destination table will execute
during the bulk copy operations. By default, no triggers
will execute. If FIRE_TRIGGERS is specified, any bulk
FIRE_TRIGGERS
load will be fully logged no matter what recovery mode
your SQL Server is in. (I'll discuss why triggers required
fully logged INSERTs in Chapter 12.)

Unlike for previous versions of SQL Server, I don't always recommend that you drop all your indexes before running
the BULK INSERT operation (or the BCP command, described in the next section). In many cases, the copy
operation doesn't perform noticeably better if a table has no indexes, so you should usually leave your indexes on a
table while copying data into it. The only exception is when you want to have multiple clients load data into the same
table simultaneously; in that case, you should drop indexes before loading the data.

When SQL Server executes a BULK INSERT operation, the rows of data go straight into the server as an OLE DB
row set. The functionality of a BULK INSERT operation is quite similar to that of the command-line utility BCP. In
contrast to the BULK INSERT operation, BCP data is sent to SQL Server as a Tabular Data Stream (TDS) result
set and flows through network protocols and Microsoft Open Data Services (ODS). This can generate a lot of
additional overhead, and you might find that BULK INSERT is up to twice as fast as BCP for copying data into a
table from a local file.

Tools and Utilities Related to Multiple-Row Inserts

INSERT and BULK INSERT aren't the only ways to get data into SQL Server tables. SQL Server also provides
some tools and utilities for loading tables from external sources. I'll mention them briefly here.

Bulk copy libraries, SQL-DMO objects, and BCP.EXE The ODBC standard doesn't directly support SQL
Server bulk copy operations. When it runs against SQL Server version 2000, however, the SQL Server 2000
ODBC driver supports a set of older DB-Library functions that perform SQL Server bulk copy operations. The
specialized bulk copy support requires that the following files be available: odbcss.h, odbcbcp.lib, and odbcbcp.dll.

The BCP.EXE command-line utility for SQL Server 2000 is written using the ODBC bulk copy interface. This utility
has little code other than the code that accepts various command-line parameters and then invokes the functions of
the bcp library. I won't go into the details of BCP.EXE except to say that it was built for function, not form. It's totally
command-line driven. If you're a fan of UNIX utilities, you'll love it. The possible arguments are similar to those for
the BULK INSERT command, and you can find a complete description in the online documentation.
This document is created with the unregistered version of CHM2PDF Pilot

The SQL-DMO BulkCopy object ultimately invokes these same bcp functions, but it provides some higher-level
methods and ease of use within the world of COM (Component Object Model) objects. The BulkCopy object
represents the parameters of a single bulk-copy command issued against SQL Server.

The BulkCopy object doesn't have a place in the SQL-DMO object tree. Instead, it is used as a parameter to the
ImportData method of the Table object and the ExportData method of the Table and View objects.

Data Transformation Services One of the most exciting tools available with SQL Server 2000 is Data
Transformation Services (DTS). Using this tool, you can import and export data to and from any OLE DB or ODBC
data source. You can do a straight copy of the data, or you can transform the data using simple SQL statements or
procedures. Alternatively, you can execute a Microsoft ActiveX script that modifies (transforms) the data when it is
copied from the source to the destination, or you can perform any operation supported by the Microsoft JScript,
PerlScript, or Microsoft Visual Basic Script (VBScript) languages.

Unlike the BCP and BULK INSERT operations, DTS creates the destination table for you as part of the copying
operation and allows you to specify the exact column names and datatypes (depending on what the destination data
store allows). DTS is installed by default with a standard SQL Server setup, and complete documentation is available
in SQL Server Books Online.

Copy SQL Server Objects SQL Server Enterprise Manager puts an easy-to-use graphical interface on top of the
SQL-DMO BulkCopy object to make it easy to transfer data between two SQL Server installations. This provides
the same functionality as the SQL Transfer Manager utility or the transfer management interface in earlier versions of
SQL Server. The tool allows you to transfer all objects—not just data, but stored procedures, rules, constraints, and
so on—but only when moving from one SQL Server database to another. The Copy SQL Server Objects tool is
accessible as a task from the DTS Package Designer.

UPDATE
UPDATE, the next data modification statement, changes existing rows. Usually, UPDATE contains a WHERE
clause that limits the update to only a subset of rows in the table. (The subset can be a single row, as would be the
case when testing for equality with the primary key values in the WHERE clause.) If no WHERE clause is provided,
UPDATE changes every row in the table. You can use the @@ROWCOUNT system function to determine the
number of rows that were updated.

Here's the basic UPDATE syntax:

UPDATE {table_name | view_name}
SET column_name1 = {expression1 | NULL | DEFAULT | (SELECT)}
[, column_name2 = {expression2 | NULL | DEFAULT | (SELECT)}
[ ,...n ]
WHERE {search_conditions}

Columns are set to an expression. The expression can be almost anything that returns a scalar value—a constant,
another column, an arithmetic expression, a bunch of nested functions that end up returning one value, a local variable,
or a system function. You can set a column to a value conditionally determined using the CASE expression or to a
value returned from a subquery. You can also set columns to NULL (if the column allows it) or to DEFAULT (if a
default is declared or if the column allows NULL), much like you can with the INSERT statement.

You can set multiple columns in a single UPDATE statement. Like a multiple-row INSERT, an UPDATE that affects
multiple rows is an atomic operation: if a single UPDATE statement affects multiple rows and one row fails a
constraint, all the changes made by that statement are rolled back.

The following simple UPDATE statements should be self-explanatory. Later in this chapter, I'll show you some that
This document is created with the unregistered version of CHM2PDF Pilot

aren't this simple and intuitive.

-- Change a specific employee's last name after his marriage
UPDATE employee
SET lname='Thomas-Kemper'
WHERE emp_id='GHT50241M'

-- Raise the price of every title by 12%

-- No WHERE clause, so it affects every row in the table
UPDATE titles
SET price=price * 1.12

-- Publisher 1389 was sold, changing its name and location.

-- All the data in other tables relating to pub_id 1389 is
-- still valid; only the name of the publisher and its
-- location have changed.
UPDATE publishers
SET pub_name='O Canada Publications',
city='Victoria',
state='BC',
country='Canada'
WHERE pub_id='1389'

-- Change the phone number of authors living in Gary, IN,

-- back to the DEFAULT value
UPDATE authors
SET phone=DEFAULT
WHERE city='Gary' AND state='IN'

Advanced UPDATE Examples

You can go well beyond these UPDATE examples, however, and use subqueries, the CASE expression, and even
joins in specifying search criteria. (Chapter 10 covers CASE in depth.) For example, the following UPDATE
statement is like a correlated subquery. It sets the ytd_sales field of the titles table to the sum of all qty fields for
matching titles:
UPDATE titles
SET titles.ytd_sales=(SELECT SUM(sales.qty) FROM sales
WHERE titles.title_id=sales.title_id)

In the next example, you can use the CASE expression with UPDATE to selectively give pay raises based on an
employee's review rating. Assume that reviews have been turned in and big salary adjustments are due. A review
rating of 4 doubles the employee's salary. A rating of 3 increases the salary by 60 percent. A rating of 2 increases the
salary by 20 percent, and a rating lower than 2 doesn't change the salary.
UPDATE employee_salaries
SET salary =
CASE review
WHEN 4 THEN salary * 2.0
WHEN 3 THEN salary * 1.6
WHEN 2 THEN salary * 1.2
ELSE salary
END

Forcing the Use of Defaults

Notice the use of the keyword DEFAULT in the last example. Sometimes you'll want to stamp a table with the name
of the user who last modified the row or with the time it was last modified. You could use system functions, such as
This document is created with the unregistered version of CHM2PDF Pilot

SUSER_SNAME or GETDATE, as the DEFAULT value, and then specify that data modification be done only via
stored procedures that explicitly update such columns to the DEFAULT keyword. Or you can make it policy that
such columns must always be set to the DEFAULT keyword in the UPDATE statement, and then you can monitor
this by also having a CHECK constraint on the same column that insists that the value be equal to that which the
function returns. (If you use GETDATE, you'll probably want to use it in conjunction with other string and date
functions to strip off the milliseconds and avoid the possibility that the value from the DEFAULT might be slightly
different than that in the CHECK.) Here's a sample using SUSER_SNAME:
CREATE TABLE update_def
(
up_id int PRIMARY KEY,
up_byname varchar(30) NOT NULL DEFAULT SUSER_SNAME()
CHECK (up_byname=SUSER_SNAME())
-- Assume other columns would be here
)
GO

UPDATE update_def
SET
-- SET other columns to their value here, and then append the
-- following
up_byname=DEFAULT
WHERE up_id=1
GO

DELETE
DELETE, the last data manipulation statement, removes rows from a table. Once the action is committed, no
undelete action is available. (If the statement is not wrapped in a BEGIN TRAN/COMMIT TRAN block, the
COMMIT, of course, occurs by default as soon as the statement completes.) Because you delete only rows, not
columns, you never specify column names in a DELETE statement as you do with INSERT or UPDATE. But in many
other ways, DELETE acts much like UPDATE does—you specify a WHERE clause to limit the DELETE to certain
rows. If you omit the WHERE clause, every row in the table will be deleted. The system function @@ROWCOUNT
keeps track of the number of rows deleted. You can delete through a view but only if the view is based on a single
table. If you delete through a view, all the underlying FOREIGN KEY constraints on the table must still be satisfied in
order for the delete to succeed.

Here's the general form of DELETE:

DELETE [FROM] {table_name | view_name} WHERE {search_conditions}

The FROM preposition is ANSI standard, but its inclusion is always optional in SQL Server (similar to INTO with
INSERT). The preposition must be specified per ANSI SQL. If the preposition is always needed, it is logically
redundant, which is why SQL Server doesn't require it.

Here are some simple examples:

DELETE discounts
-- Deletes every row from the discounts table but does not
-- delete the table itself. An empty table remains.

DELETE FROM sales WHERE qty > 5

-- Deletes those rows from the sales table that have a value for
-- qty of 6 or more

DELETE FROM WA_stores

-- Attempts to delete all rows qualified by the WA_stores view,
This document is created with the unregistered version of CHM2PDF Pilot

-- which would delete those rows from the stores table that have
-- state of WA. This delete is correctly stated but would fail
-- because of a foreign key reference.

TRUNCATE TABLE
In the first example, DELETE discounts deletes every row of that table. Every row is fully logged. SQL Server
provides a much faster way to purge all the rows from the table. TRUNCATE TABLE empties the whole table by
deallocating all the table's data pages and index pages. TRUNCATE TABLE is orders of magnitude faster than an
unqualified DELETE against the whole table. Delete triggers on the table won't fire because the rows deleted aren't
individually logged. If a foreign key references the table to be truncated, however, TRUNCATE TABLE won't work.
And if the table is publishing data for replication, which requires the log records, TRUNCATE TABLE won't work.

Despite contrary information, TRUNCATE TABLE isn't really an unlogged operation. The deletions of rows aren't
logged because they don't really occur. Rather, entire pages are deallocated. But the page deallocations are logged. If
TRUNCATE TABLE were not logged, you couldn't use it inside transactions, which must have the capacity to be
rolled back. Here's a simple example that shows that the action can indeed be rolled back:
BEGIN TRAN
-- Get the initial count of rows
SELECT COUNT(*) FROM titleauthor
25

TRUNCATE TABLE titleauthor

-- Verify that all rows are now gone

SELECT COUNT(*) FROM titleauthor
0

-- Undo the truncate operation

ROLLBACK TRAN

-- Verify that rows are still there after the undo

SELECT COUNT(*) FROM titleauthor
25

Modifying Data Through Views

You can specify INSERT, UPDATE, and DELETE statements on views as well as on tables, although you should be
aware of some restrictions and other issues. Modifications through a view end up affecting the data in an underlying
base table (and only one such table) because views don't store data. All three types of data modifications work easily
for single-table views, especially in the simplest case, in which the view exposes all the columns of the base table. But
a single-table view doesn't necessarily have to expose every column of the base table—it can restrict the view to a
subset of columns only.

If your views or your desired data modification statements don't meet the requirements for actually updating through
the view, you can create INSTEAD OF TRIGGERS on a view. The code in an INSTEAD OF TRIGGER is
executed instead of a specified modification operation, and this is the only kind of trigger you can place on a view.
(The other kind of trigger is an AFTER TRIGGER.) For example, suppose you have a view based on a join between
two tables, and you want to update columns in both tables. An INSTEAD OF TRIGGER on UPDATE of your view
can issue two separate UPDATE statements so that both tables can actually be updated. I'll discuss INSTEAD OF
TRIGGERS along with AFTER TRIGGERS in Chapter 12. The remaining discussion in this section will assume there
are no INSTEAD OF TRIGGERS on your views and that you want to modify the base table data directly through the
view.
This document is created with the unregistered version of CHM2PDF Pilot

For any direct modification, all the underlying constraints of the base table must still be satisfied. For example, if a
column in the base table is defined as NOT NULL and doesn't have a DEFAULT declared for it, the column must be
visible to the view and the insert must supply a value for it. If the column weren't part of the view, an insert through
that view could never work because the NOT NULL constraint could never be satisfied. And, of course, you can't
modify or insert a value for a column that's derived by the view, such as an arithmetic calculation or concatenation of
two strings of the base table. (You can still modify the nonderived columns in the view, however.)

Basing a view on multiple tables is far less straightforward. You can issue an UPDATE statement against a view that
is a join of multiple tables, but only if all columns being modified (that is, the columns in the SET clause) are part of the
same base table. An INSERT statement can also be performed against a view that does a join only if columns from a
single table are specified in the INSERT statement's column list. Only the table whose columns are specified will have
a row inserted: any other tables in the view will be unaffected. A DELETE statement can't be executed against a view
that's a join because entire rows are deleted and modifications through a view can affect only one base table. Because
no columns are specified in a delete, which table would the rows be deleted from?

In the real world, you would probably have little use for an INSERT statement against a view with a join because all
but one table in the underlying query would be totally ignored. But the insert is possible. The following simple example
illustrates this:
CREATE TABLE one
(
col11 int NOT NULL,
col12 int NOT NULL
)

CREATE TABLE two

(
col21 int NOT NULL,
col22 int NOT NULL
)
GO

CREATE VIEW one_two

AS
(SELECT col11, col12, col21, col22
FROM one LEFT JOIN two ON (col11=col21))
GO

INSERT one_two (col11, col12)

VALUES (1, 2)

SELECT * FROM one_two

Here's the result:

col11 col12 col21 col22
----- ----- ----- -----
1 2 NULL NULL

Notice that this insert specifies values only for columns from table one, and only table one gets a new row. Selecting
from the view produces the row only because LEFT OUTER JOIN is specified in the view. Because table two
contains no actual rows, a simple equijoin would have found no matches. Although the row would still have been
inserted into table one, it would have seemingly vanished from the view. You could specify the view as an equijoin
and use WITH CHECK OPTION to prevent an insert that wouldn't find a match. But the insert must still affect only
one of the tables, so matching rows would have to already exist in the other table. I'll come back to WITH CHECK
OPTION in the next section; for now, here's how it would be specified:
CREATE VIEW one_two_equijoin
This document is created with the unregistered version of CHM2PDF Pilot

AS
(SELECT col11, col12, col21, col22
FROM one JOIN two ON (col11=col21))
WITH CHECK OPTION
GO

If you try to specify all columns with either view formulation, even if you simply try to insert NULL values into the
columns of table two, an error results because the single INSERT operation can't be performed on both tables.
Admittedly, the error message is slightly misleading.
INSERT one_two (col11, col12, col21, col22)
VALUES (1, 2, NULL, NULL)

Server: Msg 4405, Level 16, State 2, Line 1

View or function 'one_two' is not updatable because the modification
affects multiple base tables..

Similarly, a DELETE against this view with a join would be disallowed and results in the same message:
DELETE one_two

Server: Msg 4405, Level 16, State 1, Line 1

View or function 'one_two' is not updatable because the modification
affects multiple base tables referenced.

The UPDATE case isn't common, but it's somewhat more realistic. You'll probably want to avoid allowing updates
through views that do joins (unless you have INSTEAD OF TRIGGERS), and the next example shows why.

Given the following view,

CREATE VIEW titles_and_authors
AS
(
SELECT A.au_id, A.au_lname, T.title_id, T.title
FROM
authors AS A
FULL OUTER JOIN titleauthor AS TA ON (A.au_id=TA.au_id)
FULL OUTER JOIN titles AS T ON (TA.title_id=T.title_id)
)

selecting from the view

SELECT * FROM titles_and_authors

yields this:
au_id au_lname title_id title
----------- ------------- -------- ------------------------------
172-32-1176 White PS3333 Prolonged Data Deprivation:
Four Case Studies
213-46-8915 Green BU1032 The Busy Executive's Database
Guide
213-46-8915 Green BU2075 You Can Combat Computer
Stress!
238-95-7766 Carson PC1035 But Is It User Friendly?
267-41-2394 O'Leary BU1111 Cooking with Computers:
Surreptitious Balance Sheets
This document is created with the unregistered version of CHM2PDF Pilot

267-41-2394 O'Leary TC7777 Sushi, Anyone?

274-80-9391 Straight BU7832 Straight Talk About Computers
409-56-7008 Bennet BU1032 The Busy Executive's Database
Guide
427-17-2319 Dull PC8888 Secrets of Silicon Valley
472-27-2349 Gringlesby TC7777 Sushi, Anyone?
486-29-1786 Locksley PC9999 Net Etiquette
486-29-1786 Locksley PS7777 Emotional Security: A New
Algorithm
648-92-1872 Blotchet-Halls TC4203 Fifty Years in Buckingham
Palace Kitchens
672-71-3249 Yokomoto TC7777 Sushi, Anyone?
712-45-1867 del Castillo MC2222 Silicon Valley Gastronomic
Treats
722-51-5454 DeFrance MC3021 The Gourmet Microwave
724-80-9391 MacFeather BU1111 Cooking with Computers:
Surreptitious Balance Sheets
724-80-9391 MacFeather PS1372 Computer Phobic AND Non-Phobic
Individuals: Behavior
Variations
756-30-7391 Karsen PS1372 Computer Phobic AND Non-Phobic
Individuals: Behavior
Variations
807-91-6654 Panteley TC3218 Onions, Leeks, and Garlic:
Cooking Secrets of the
Mediterranean
846-92-7186 Hunter PC8888 Secrets of Silicon Valley
899-46-2035 Ringer MC3021 The Gourmet Microwave
899-46-2035 Ringer PS2091 Is Anger the Enemy?
998-72-3567 Ringer PS2091 Is Anger the Enemy?
998-72-3567 Ringer PS2106 Life Without Fear
341-22-1782 Smith NULL NULL
527-72-3246 Greene NULL NULL
724-08-9931 Stringer NULL NULL
893-72-1158 McBadden NULL NULL
NULL NULL MC3026 The Psychology of Computer
Cooking
(30 rows affected)

The following UPDATE statement works fine because only one underlying table, authors, is affected. This example
changes the author's name from DeFrance to DeFrance-Macy.
UPDATE TITLES_AND_AUTHORS
SET au_lname='DeFrance-Macy'
WHERE au_id='722-51-5454'
(1 row(s) affected)

This UPDATE statement yields an error, however, because two tables from a view can't be updated in the same
statement:
UPDATE TITLES_AND_AUTHORS
SET au_lname='DeFrance-Macy', title='The Gourmet Microwave Cookbook'
WHERE au_id='722-51-5454' and title_id='MC3021'

Server: Msg 4405, Level 16, State 2, Line 1

View or function 'TITLES_AND_AUTHORS' is not updatable because the modification affects
multiple base tables referenced.

If you created the view, it might seem obvious that the UPDATE statement above won't be allowed. But if the person
doing the update isn't aware of the underlying tables—which a view does a great job of hiding—it will not be obvious
This document is created with the unregistered version of CHM2PDF Pilot

why one UPDATE statement works and the other one doesn't.

Views that join multiple tables can be extremely useful for querying and are wonderful constructs that make querying
simpler and less prone to bugs from misstated joins. But modifications against them are problematic. Even if you have
created INSTEAD OF TRIGGERS, you still need to carefully plan exactly what the INSTEAD OF TRIGGER will
do. If you do allow an update through a view that's a join, be sure that the relationships are properly protected via
FOREIGN KEY constraints or triggers; otherwise, you'll run into bigger problems.

Suppose we want to change only the au_id column in the example view above. If the existing value in the underlying
authors table is referenced by a row in titleauthor, such an update won't be allowed because it would orphan the
row in titleauthor. The FOREIGN KEY constraint protects against the modification. But here, we'll temporarily
disable the FOREIGN KEY constraint between titleauthor and titles:
ALTER TABLE titleauthor -- This disables the constraint
NOCHECK CONSTRAINT ALL

Notice that although we'll be updating the authors table, the constraint we're disabling (the constraint that would
otherwise be violated) is on the titleauthor table. New users often forget that a FOREIGN KEY constraint on one
table is essentially a constraint on both the referencing table (here, titleauthor) and the referenced table (here, titles).
With the constraint now disabled, we'll change the value of au_id through the view for the author Anne Ringer:
-- With constraint now disabled, the following update succeeds:
UPDATE titles_and_authors
SET au_id='111-22-3333'
WHERE au_id='899-46-2035'

But look at the effect on the same SELECT of all rows in the view:
au_id au_lname title_id title
----------- ------------- -------- ------------------------------
172-32-1176 White PS3333 Prolonged Data Deprivation:
Four Case Studies
213-46-8915 Green BU1032 The Busy Executive's Database
Guide
213-46-8915 Green BU2075 You Can Combat Computer
Stress!
238-95-7766 Carson PC1035 But Is It User Friendly?
267-41-2394 O'Leary BU1111 Cooking with Computers:
Surreptitious Balance Sheets
267-41-2394 O'Leary TC7777 Sushi, Anyone?
274-80-9391 Straight BU7832 Straight Talk About Computers
409-56-7008 Bennet BU1032 The Busy Executive's Database
Guide
427-17-2319 Dull PC8888 Secrets of Silicon Valley
472-27-2349 Gringlesby TC7777 Sushi, Anyone?
486-29-1786 Locksley PC9999 Net Etiquette
486-29-1786 Locksley PS7777 Emotional Security: A New
Algorithm
648-92-1872 Blotchet-Halls TC4203 Fifty Years in Buckingham
Palace Kitchens
672-71-3249 Yokomoto TC7777 Sushi, Anyone?
712-45-1867 del Castillo MC2222 Silicon Valley Gastronomic
Treats
722-51-5454 DeFrance-Macy MC3021 The Gourmet Microwave
724-80-9391 MacFeather BU1111 Cooking with Computers:
Surreptitious Balance Sheets
724-80-9391 MacFeather PS1372 Computer Phobic AND Non-Phobic
Individuals: Behavior
Variations
This document is created with the unregistered version of CHM2PDF Pilot

756-30-7391 Karsen PS1372 Computer Phobic AND Non-Phobic

Individuals: Behavior
Variations
807-91-6654 Panteley TC3218 Onions, Leeks, and Garlic:
Cooking Secrets of the
Mediterranean
846-92-7186 Hunter PC8888 Secrets of Silicon Valley
NULL NULL MC3021 The Gourmet Microwave
NULL NULL PS2091 Is Anger the Enemy?
998-72-3567 Ringer PS2091 Is Anger the Enemy?
998-72-3567 Ringer PS2106 Life Without Fear
111-22-3333 Ringer NULL NULL
341-22-1782 Smith NULL NULL
527-72-3246 Greene NULL NULL
724-08-9931 Stringer NULL NULL
893-72-1158 McBadden NULL NULL
NULL NULL MC3026 The Psychology of Computer
Cooking
(31 rows affected)

Although we haven't added or deleted any rows, the view now produces 31 rows instead of 30 rows, as was the
case earlier, because the outer join fabricates a row for the new au_id that has no match in the titles table. (It
appears with NULL title fields.) Also notice that the two titles that matched the old au_id now have NULLs in their
author fields. This might be comprehensible to someone who has a detailed understanding of the tables, but if the view
is the only window for updating the table, it will be extremely confusing.

Even if your FOREIGN KEY constraint specifies that any updates are to be cascaded to the referencing tables,
you'll still have the problem of affecting columns from more than one table without knowing which is which. In the
example above, both the authors and titleauthor table have an au_id column, so we would have to make sure we
knew which table was actually being updated through the view.

WITH CHECK OPTION

Earlier, you saw that modifying data with views based on one table is pretty straightforward—but there's one
"gotcha" to think about: disappearing rows. By default, a view can allow an UPDATE or an INSERT even if the result
is that the row no longer qualifies for the view. Consider the following view, which qualifies only stores in the state of
Washington:
CREATE VIEW WA_stores AS
SELECT * FROM stores
WHERE state='WA'
GO

SELECT stor_id, stor_name, state

FROM WA_stores

Here's the result:

stor_id stor_name state
------- ------------------------------------ -----
6380 Eric the Read Books WA
7131 Doc-U-Mat: Quality Laundry and Books WA
(2 rows affected)

The following UPDATE statement changes both qualifying rows so that they no longer satisfy the view's criteria and
have seemingly disappeared:
This document is created with the unregistered version of CHM2PDF Pilot

UPDATE WA_stores
SET state='CA'
SELECT stor_id, stor_name, state
FROM WA_stores

Here's the result:

stor_id stor_name state
------- --------- -----
(0 rows affected)

Modifications you make against a view without WITH CHECK OPTION can result in rows being added or
modified in the base table, but the rows can't be selected from the view because they don't meet the view's criteria.
The rows seem to disappear, as the example above illustrates. For users who realize that WA_stores is a view and
who understand exactly how views work, this might not be a big problem. But if your environment attempts to have
users treat views as if they were actually tables, having an UPDATE statement cause rows to disappear can be very
disconcerting.

If you allow data to be modified through a view that uses a WHERE clause and you want your views to act exactly
as if they were tables, consider using WITH CHECK OPTION when you define the view. In fact, I think it would be
better if WITH CHECK OPTION were the default behavior. But the behavior described above is in accordance
with the ANSI and ISO SQL standards, and the expected result is that the disappearing row phenomenon can occur
unless WITH CHECK OPTION is specified. Of course, any modifications you make through the view must also
satisfy the constraints of the underlying table or else the statement will be aborted. This is true whether or not WITH
CHECK OPTION is declared.

The following example shows how WITH CHECK OPTION protects against the disappearing rows phenomenon:
CREATE VIEW WA_stores AS
SELECT * FROM stores
WHERE state='WA'
WITH CHECK OPTION
GO

UPDATE WA_stores
SET state='CA'

Here's the result:

Server: Msg 550, Level 16, State 1, Line 1
The attempted insert or update failed because the target view either
specifies WITH CHECK OPTION or spans a view that specifies WITH CHECK
OPTION and one or more rows resulting from the operation did not
qualify under the CHECK OPTION constraint.

The statement has been terminated.

SELECT stor_id, stor_name, state

FROM WA_stores

And here's the final result:

stor_id stor_name state
------- ------------------------------------ -----
6380 Eric the Read Books WA
7131 Doc-U-Mat: Quality Laundry and Books WA
(2 rows affected)
This document is created with the unregistered version of CHM2PDF Pilot

Reenabling Integrity Constraints

To demonstrate a point in the preceding section, we disabled foreign key checks on the titleauthor table. When
you're reenabling constraints, make sure that no rows violate the constraints. By default, when you add a new
constraint, SQL Server automatically checks for violations, and the constraint won't be added until the data is cleaned
up and such violations are eliminated. You can suppress this check by using the NOCHECK option. With this option,
when you reenable an existing constraint that has just been disabled, SQL Server doesn't recheck the constraint's
relationships.

After reenabling a disabled constraint, adding a new constraint with the NOCHECK option, or performing a bulk
load operation without specifying the CHECK CONSTRAINTS option, you should check to see that constraints
have been satisfied.

You can query for constraint violations using subqueries of the type discussed in Chapter 7. You must formulate a
separate query to check for every constraint, which can be tedious if you have many constraints. Alternatively, you
can execute DBCC CHECKCONSTRAINTS, as you'll see shortly.

In an earlier example, we disabled the constraints on the titleauthor table and updated one row in such a way that
the FOREIGN KEY constraint was violated. Here's an example that reenables the constraints on titleauthor and
then runs DBCC CHECKCONSTRAINTS, which reveals the constraint failure:
-- Reenable constraints. Note that this does not check the validity
-- of the constraints.
ALTER TABLE titleauthor
CHECK CONSTRAINT ALL
GO

DBCC CHECKCONSTRAINTS (titleauthor)

RESULT:

Table Constraint Where

------------ ------------------------------- ----------------------
titleauthor FK__titleauth__au_id__07020F21 au_id = '899-46-2035'

The constraint failure occurred because of the foreign key reference to the authors table. The output shows which
value(s) in titleauthor has no matching value in authors. In earlier versions of SQL Server, you could actually issue a
dummy UPDATE statement that set the column with the constraint to itself, as in this example:
UPDATE titleauthor
SET au_id = au_id

This would cause SQL Server to verify constraints on the au_id column and detect that there was an unmatched
foreign key. However, in SQL Server 2000, the optimizer will detect that this operation is really not doing anything,
so no plan is generated, no UPDATE is carried out, and no constraints are checked. DBCC
CHECKCONSTRAINTS performs all the validation you need.
This document is created with the unregistered version of CHM2PDF Pilot

34

Data Modification Internals

In Chapter 6, we examined how SQL Server stores data. Now we'll look at what SQL Server actually does
internally when data is modified. We've already discussed (in Chapters 3 and 8) clustered indexes and how they
control space usage in SQL Server. As a rule of thumb, you should always have a clustered index on a table. Only a
couple of cases exist in which you would not have one. In Chapter 15, when we look at the query optimizer, we'll
examine the benefits and tradeoffs of clustered and nonclustered indexes and review some guidelines for their use. In
this chapter, we'll look only at how SQL Server deals with the existence of indexes when processing data
modification statements.

Inserting Rows
When inserting a new row into a table, SQL Server must determine where to put it. When a table has no clustered
index—that is, when the table is a heap—a new row is always inserted wherever room is available in the table. In
Chapter 5, I showed you how IAMs and the PFS pages keep track of which extents in a file already belong to a table
and which of the pages in those extents have space available. Even without a clustered index, space management is
quite efficient. If no pages with space are available, SQL Server must allocate a whole new extent to the table.
Chapter 5 also discussed how the GAMs and SGAMs were used to find extents available to be allocated to an
object.

A clustered index directs an insert to a specific page based on the value the new row has for the clustered index key
columns. The insert occurs when the new row is the direct result of an INSERT statement or when it's the result of an
UPDATE statement executed via a delete-followed-by-insert (delete/insert) strategy. New rows are inserted into their
clustered position, splicing in a page via a page split if the current page has no room. You saw in Chapters 3 and 6
that if a clustered index isn't declared as unique and duplicate values are inserted for a key, an automatic uniqueifier is
generated for all subsequent rows with the same key. Internally, SQL Server treats all clustered index keys as unique.

Because the clustered index dictates a particular ordering for the rows in a table, every new row has a specific
location where it belongs. If there's no room for the new row on the page where it belongs, you must allocate a new
page and link it into the list of pages. If possible, this new page is allocated from the same extent as the other pages to
which it will be linked. If the extent is full, a new extent (eight pages, or 64 KB) is allocated to the object. As
described in Chapter 5, SQL Server uses the GAM pages to find an available extent.

Splitting Pages
After SQL Server finds the new page, the original page must be split; half the rows are left on the original page, and
half are moved to the new page. In some cases, SQL Server finds that even after the split, there's no room for the
new row, which, because of variable-length fields, could potentially be much larger than any of the existing rows on
the pages. After the split, one or two rows are promoted to the parent page.

An index tree is always searched from the root down, so during an insert operation it is split on the way down. This
means that while the index is being searched on an insert, the index is being protected in anticipation of possibly being
updated. A parent node (not a leaf node) is latched until the child node is known to be available for its own latch.
Then the parent latch can be released safely.

Before the latch on a parent node is released, SQL Server determines whether the page will accommodate another
two rows; if not, it splits the page. This occurs only if the page is being searched with the objective of adding a row to
the index. The goal is to ensure that the parent page always has room for the row or rows that result from a child page
This document is created with the unregistered version of CHM2PDF Pilot

splitting. (Occasionally, this results in pages being split that don't need to be—at least not yet. In the long run, it's a
performance optimization.) The type of split depends on the type of page being split: a root page of an index, an
intermediate index page, or a data page.

Splitting the Root Page of an Index

If the root page of an index needs to be split in order for a new index row to be inserted, two new pages are
allocated to the index. All the rows from the root are split between these two new pages, and the new index row is
inserted into the appropriate place on one of these pages. The original root page is still the root, but now it has only
two rows on it, pointing to each of the newly allocated pages. A root page split creates a new level in the index.
Because indexes are usually only a few levels deep, this type of split doesn't occur often.

Splitting the Intermediate Index Page

An intermediate index page split is accomplished simply by locating the midpoint of the index keys on the page,
allocating a new page, and then copying the lower half of the old index page into the new page. Again, this doesn't
occur often, although it's more common than splitting the root page.

Splitting the Data Page

A data page split is the most interesting and potentially common case, and it's probably the only split that you, as a
developer, should be concerned with. Data pages split only under insert activity and only when a clustered index
exists on the table. If no clustered index exists, the insert goes on any page that has room for the new row, according
to the PFS pages. Although splits are caused only by insert activity, that activity can be a result of an UPDATE
statement, not just an INSERT statement. As you're about to learn, if the row can't be updated in place or at least on
the same page, the update is performed as a delete of the original row followed by an insert of the new version of the
row. The insertion of the new row can, of course, cause a page split.

Splitting a data page is a complicated operation. Much like an intermediate index page split, it's accomplished by
locating the midpoint of the index keys on the data page, allocating a new page, and then copying half of the old page
into the new page. It requires that the index manager determine the page on which to locate the new row and then
handle large rows that don't fit on either the old page or the new page. When a data page is split, the clustered index
key values don't change, so the nonclustered indexes aren't affected.

Let's look at what happens to a page when it splits. The following script creates a table with large rows—so large, in
fact, that only five rows will fit on a page. Once the table is created and populated with five rows, we'll find its first
page by looking at the sysindexes.first column. As discussed in Chapter 6, I'll convert the address of the first page of
the table to a decimal value and then use DBCC PAGE to look at the contents of the page. Because we don't need to
see all 8020 bytes of data on the page, we'll look at only the row offset array at the end of the page and then see what
happens to those rows when we insert a sixth row.
/* First create the table */
USE pubs
GO

DROP TABLE bigrows

GO

CREATE TABLE bigrows

(
a int primary key,
b varchar(1600)
)
GO

/* Insert five rows into the table */

This document is created with the unregistered version of CHM2PDF Pilot

INSERT INTO bigrows

VALUES (5, REPLICATE('a', 1600))

INSERT INTO bigrows

VALUES (10, replicate('b', 1600))

INSERT INTO bigrows

VALUES (15, replicate('c', 1600))

INSERT INTO bigrows

VALUES (20, replicate('d', 1600))

INSERT INTO bigrows

VALUES (25, replicate('e', 1600))
GO

SELECT first FROM sysindexes WHERE id = OBJECT_ID ('bigrows')

GO

/* Convert the value returned to a decimal by byte-swapping

and then doing a hex-to-decimal conversion on the last four
bytes. See Chapter 6 for details. For our example,
we'll assume the converted value is decimal 249. */

DBCC TRACEON(3604)
GO

DBCC PAGE(pubs, 1, 249, 1, 1)

GO

Here is the row offset array from the results:

Row - Offset
4 (0x4) - 6556 (0x199c)
3 (0x3) - 4941 (0x134d)
2 (0x2) - 3326 (0xcfe)
1 (0x1) - 1711 (0x6af)
0 (0x0) - 96 (0x60)

Now we'll insert one more row and look at the row offset array again:
INSERT INTO bigrows
VALUES (22, REPLICATE('x', 1600))
GO

DBCC PAGE(pubs, 1, 249, 1, 1)

GO

When you inspect the original page after the split, you might find that it contains either the first half of the rows from
the original page or the second half. SQL Server normally moves the rows so that the new row to be inserted goes on
the new page. Since the rows are moving anyway, it makes more sense to adjust their positions to accommodate the
new inserted row. In this example, the new row, with a clustered key value of 22, would have been inserted in the
second half of the page. So when the page split occurs, the first three rows stay on the original page, 249. You need
to inspect the page header to find the location of the next page, which contains the new row.

The page number is indicated by the m_nextPage field. This value is expressed as a file number:page number pair,
in decimal, so you can easily use it with the DBCC PAGE command. When I ran this query, I got an m_nextPage of
1:2592, so I ran the following command:
This document is created with the unregistered version of CHM2PDF Pilot

DBCC PAGE(pubs, 1, 2592, 1, 1)

Here's the row offset array after the insert for the second page:
Row - Offset
2 (0x2) - 1711 (0x6af)
1 (0x1) - 3326 (0xcfe)
0 (0x0) - 96 (0x60)

Note that after the page split, three rows are on the page: the last two original rows with keys of 20 and 25, and the
new row with a key of 22. If you examine the actual data on the page, you'll notice that the new row is at slot position
1, even though the row itself is the last one on the page. Slot 1 (with value 22) starts at offset 3326, and slot 2 (with
value 25) starts at offset 1711. The clustered key ordering of the rows is indicated by the slot number of the row, not
by the physical position on the page. If a clustered index is on the table, the row at slot 1 will always have a key value
less than the row at slot 2 and greater than the row at slot 0.

Although the typical page split isn't too expensive, you'll want to minimize the frequency of page splits in your
production system, at least during peak usage times. You can avoid system disruption during busy times by reserving
some space on pages using the FILLFACTOR clause when you're creating the clustered index on existing data. You
can use this clause to your advantage during your least busy operational hours by periodically re-creating the clustered
index with the desired FILLFACTOR. That way, the extra space is available during peak usage times, and you'll save
the overhead of splitting then. If there are no "slow" times, you can use DBCC INDEXDEFRAG to reorganize an
index's pages and readjust the FILLFACTOR without having to make the entire table unavailable. Using SQL Server
Agent, you can easily schedule the rebuilding of indexes or the execution of DBCC INDEXDEFRAG to occur at
times when activity is lowest.
NOTE
The FILLFACTOR setting is helpful only when you're creating the index on existing data; the FILLFACTOR value
isn't maintained when you insert new data. Trying to maintain extra space on the page would actually defeat the
purpose because you'd need to split pages anyway to keep that amount of space available.

Deleting Rows
When you delete rows from a table, you have to consider what happens both to the data pages and the index pages.
Remember that the data is actually the leaf level of a clustered index, and deleting rows from a table with a clustered
index happens the same way as deleting rows from the leaf level of a nonclustered index. Deleting rows from a heap is
managed in a different way, as is deleting from node pages of an index.

Deleting Rows from a Heap

SQL Server 2000 doesn't automatically compress space on a page when a row is deleted. As a performance
optimization, the compaction doesn't occur until a page needs additional contiguous space for inserting a new row.
You can see this in the following example, which deletes a row from the middle of a page and then inspects that page
using DBCC PAGE. Figure 9-1 shows the output from DBCC PAGE.
USE pubs
GO

DROP TABLE smallrows

GO

CREATE TABLE smallrows

(
a int identity,
b char(10)
This document is created with the unregistered version of CHM2PDF Pilot

)
GO

INSERT INTO smallrows

VALUES ('row 1')
INSERT INTO smallrows
VALUES ('row 2')
INSERT INTO smallrows
VALUES ('row 3')
INSERT INTO smallrows
VALUES ('row 4')
INSERT INTO smallrows
VALUES ('row 5')
GO

SELECT first FROM sysindexes WHERE id = OBJECT_ID ('smallrows')

GO

DBCC TRACEON(3604)
GO

DBCC PAGE(pubs, 1, 248, 1, 1)

GO

Figure 9-1. Page contents before a delete.

Now we'll delete the middle row (WHERE a = 3) and look at the page again. Figure 9-2 contains the output from
the second execution of DBCC PAGE.
This document is created with the unregistered version of CHM2PDF Pilot

DELETE FROM smallrows

WHERE a = 3
GO

DBCC PAGE(pubs, 1, 248, 1, 1)

GO

Figure 9-2. Page contents after the delete from a heap.

Note that in the heap in Figure 9-2, the row doesn't show up in the page itself. The row offset array at the bottom of
the page shows that row 3 (at slot 2) is now at offset 0 (which means there really is no row using slot 2), and the row
using slot 3 is at its same offset as before the delete. The data on the page is not compressed.

Deleting Rows from a B-Tree

In the leaf level of an index, when rows are deleted, they're marked as ghost records. This means that the row stays
on the page but a bit is changed in the row header to indicate that the row is really a ghost. The page header also
reflects the number of ghost records on a page. Ghost records are primarily a concurrency optimization for key-range
locking. I'll discuss the details of key-range locking, as well as all other locking modes, in Chapter 14, but a short
explanation is in order here, just for completeness.

Ghost records are present only in the index leaf nodes. If ghost records weren't used, the entire range surrounding a
deleted key would have to be locked. So suppose you have a unique index on an integer and the index contains the
values 1, 30, and 100. If you delete 30, SQL Server will need to lock (and prevent inserts into) the entire range
between 1 and 100. With ghosted records, the 30 is still visible to be used as an endpoint of a key-range lock so that
during the delete transaction, SQL Server can allow inserts for any value other than 30 to proceed.

SQL Server provides a special housekeeping thread that periodically checks B-trees for ghosted records and
asynchronously removes them from the leaf level of the index. This same thread carries out the automatic shrinking of
databases if you have that option set. When this thread is active, the Current Activity window in SQL Server
Enterprise Manager or the sp_who command will show you an active process with a spid value of 6.
This document is created with the unregistered version of CHM2PDF Pilot

The following example builds the same table used in the previous DELETE example, but this time the table has a
primary key declared, which means a clustered index will be built. The data is the leaf level of the clustered index, so
when the row is removed, it will be marked as a ghost. (Figure 9-3 shows the output from DBCC PAGE.)
USE pubs
GO

DROP TABLE smallrows

GO

CREATE TABLE smallrows

(
a int IDENTITY PRIMARY KEY,
b char(10)
)
GO

INSERT INTO smallrows

VALUES ('row 1')
INSERT INTO smallrows
VALUES ('row 2')
INSERT INTO smallrows
VALUES ('row 3')
INSERT INTO smallrows
VALUES ('row 4')
INSERT INTO smallrows
VALUES ('row 5')
GO

SELECT first FROM sysindexes WHERE id = OBJECT_ID ('smallrows')

GO

DELETE FROM smallrows

WHERE a = 3
GO

DBCC PAGE(pubs, 1, 253, 1, 1)

GO
This document is created with the unregistered version of CHM2PDF Pilot

Figure 9-3. Page contents after the delete from a table with a clustered index.

Note that in the figure, the row still shows up in the page itself because the table has a clustered index. The header
information for the row shows that this is really a ghosted record. The row offset array at the bottom of the page
shows that the row at slot 2 is still at the same offset and that all rows are in the same location as before the deletion.
In addition, the page header gives us a value (m_ghostRecCnt) for the number of ghosted records in the page. To
see the total count of ghost records in a table, you can enable trace flag 2514. If you then execute the DBCC
CHECKTABLE command, you'll see the number of ghost records in that table.
DBCC TRACEON (2514)
GO

DBCC CHECKTABLE (smallrows)

You'll see results that look like this:

DBCC results for 'smallrows'.
There are 4 rows in 1 page for object 'smallrows'.
Ghost Record count = 1
DBCC execution completed. If DBCC printed error messages, contact your
system administrator.
This document is created with the unregistered version of CHM2PDF Pilot

Note that the trace flag setting applies only to the current connection. You can force the setting to apply to all
connections by enabling the trace flag along with a -1:
DBCC TRACEON (2514, -1)

Also note that I put the DELETE statement inside a user transaction. The housekeeping thread that cleans up ghost
records is fast and efficient, and there is a chance that the ghost record will be cleaned up between the time that I
issue the DELETE operation and the time I execute DBCC PAGE to look at the rows. To guarantee that the ghost
will not be cleaned up, I put the DELETE into a transaction, and I don't commit or roll back the transaction before
examining the page or running DBCC CHECKTABLE. The housekeeping thread will not clean up ghost records that
are part of an active transaction.

Deleting Rows in the Node Levels of an Index

When you delete a row from a table, all nonclustered indexes need to be maintained because every nonclustered
index has a pointer to the row that's now gone. Rows in index node pages aren't ghosted when deleted, but just as
with heap pages, the space isn't compressed until new index rows need space in that page.

Reclaiming Pages
When the last row is deleted from a data page, the entire page is deallocated. (If the page is the only one remaining in
the table, it isn't deallocated. A table always contains at least one page, even if it's empty.) This also results in the
deletion of the row in the index page that pointed to the old data page. Index pages are deallocated if an index row is
deleted (which, again, might occur as part of a delete/insert update strategy), leaving only one entry in the index page.
That entry is moved to its neighboring page, and then the empty page is deallocated.

The discussion so far has focused on the page manipulation necessary for deleting a single row. If multiple rows are
deleted in a single deletion operation, you need to be aware of some other issues. Because the issues of modifying
multiple rows in a single query are the same for inserts, updates, and deletes, we'll discuss this issue near the end of
the chapter.

Updating Rows
SQL Server updates rows in multiple ways, automatically and invisibly choosing the fastest update strategy for the
specific operation. In determining the strategy, SQL Server evaluates the number of rows affected, how the rows will
be accessed (via a scan or an index retrieval, and via which index), and whether changes to the index keys will occur.
Updates can happen either in place or as a delete followed by an insert. In addition, updates can be managed by the
query processor or by the storage engine. In this section, we'll examine only whether the update happens in place. The
question of whether the update is controlled by the query processor or the storage engine is actually relevant to all
data modification operations (not just updates), so we'll look at that in a separate section.

Prior to version 7, SQL Server could perform either a direct or a deferred update. Deferred meant that the update
occurred in two complete passes. The first pass would use the transaction log as a holding area, in which SQL Server
recorded all the changes that were going to be made and marked them as NO-OP because they weren't actually
executed at that time. The second pass would then reread the log and apply the changes. Because all
changes—including changes to every nonclustered index—must be recorded twice, this was log-intensive. In addition,
any time a row moved, every nonclustered index had to be updated to hold the new location of the row. Deferred
updates were by far the slowest kind of update.

SQL Server 2000 doesn't have deferred updates. All updates are done in a direct mode, without using the
transaction log as an intermediate holding area. In addition, because of the way SQL Server 2000 maintains
This document is created with the unregistered version of CHM2PDF Pilot

nonclustered indexes, the overhead of moving rows, even if the table has multiple nonclustered indexes, is an
inexpensive operation.

Moving Rows
What happens if a table row has to move to a new location? In SQL Server 2000, this can happen because a row
with variable-length columns is updated to a new, larger size so that it no longer fits on the original page. It can also
happen when the clustered index column is changing, because rows are stored in order of the clustering key. For
example, if we have a clustered index on lastname, a row with a lastname value of Abbot will be stored near the
beginning of the table. If the lastname value is then updated to Zappa, this row will have to move to near the end of
the table.

In Chapter 8, we looked at the structure of indexes and saw that the leaf level of nonclustered indexes contains a row
locator for every single row in the table. If the table has a clustered index, that row locator is the clustering key for that
row. So if—and only if—the clustered index key is being updated, modifications are required in every nonclustered
index. Keep this in mind when you decide which columns to build your clustered index on. It's a great idea to cluster
on a nonvolatile column.

If a row moves because it no longer fits on the original page, it will still have the same row locator (in other words,
the clustering key for the row stays the same), and no nonclustered indexes will have to be modified.

In Chapter 8, you also saw that if a table has no clustered index (in other words, it's a heap), the row locator stored
in the nonclustered index is actually the physical location of the row. In SQL Server 2000, if a row in a heap moves to
a new page, the row will leave a forwarding pointer in the original location. The nonclustered indexes won't need to be
changed; they'll still refer to the original location, and from there they'll be directed to the new location.

Let's look at an example. I'll create a table a lot like the one we created for doing inserts, but this table will have a
third column of variable length. After I populate the table with five rows, which will fill the page, I'll update one of the
rows to make its third column much longer. The row will no longer fit on the original page and will have to move.
Remember that I need to select the first column from sysindexes to know the address of the page. The value you
actually supply to DBCC PAGE depends on the value that first returns.
USE pubs
GO

DROP TABLE bigrows

GO

CREATE TABLE bigrows

(
a int IDENTITY ,
b varchar(1600),
c varchar(1600))
GO

INSERT INTO bigrows

VALUES (REPLICATE('a', 1600), '')
INSERT INTO bigrows
VALUES (REPLICATE('b', 1600), '')
INSERT INTO bigrows
VALUES (REPLICATE('c', 1600), '')
INSERT INTO bigrows
VALUES (REPLICATE('d', 1600), '')
INSERT INTO bigrows
VALUES (REPLICATE('e', 1600), '')
GO

SELECT first FROM sysindexes WHERE id = OBJECT_ID ('bigrows')

GO
This document is created with the unregistered version of CHM2PDF Pilot

UPDATE bigrows
SET c = REPLICATE('x', 1600)
WHERE a = 3
GO

DBCC TRACEON(3604)
GO

DBCC PAGE(pubs, 1, 138, 1, 1)

GO

I won't show you the entire output from the DBCC PAGE command, but I'll show you what appears in the slot
where the row with a = 3 formerly appeared:
Slot 2, Offset 0x1feb
---------------------
Record Type = FORWARDING_STUB Record Attributes =
1d9cdfeb: 00008c04 00000100 00 .........

The value of 4 in the first byte means that this is just a forwarding stub. The 00008c in the next 3 bytes is the page
number to which the row has been moved. Because this is a hexadecimal value, we need to convert it to 138 decimal.
The next group of 4 bytes tells us that the page is at slot 0, file 1. If you then use DBCC PAGE to look at that page
138, you can see what the forwarded record looks like.

Managing Forward Pointers

Forward pointers allow you to modify data in a heap without worrying about having to make drastic changes to the
nonclustered indexes. If a row that has been forwarded must move again, the original forwarding pointer is updated to
point to the new location. You'll never end up with a forwarding pointer pointing to another forwarding pointer. In
addition, if the forwarded row shrinks enough to fit in its original place, the forward pointer is eliminated and the
record moves back to its place.

A future version of SQL Server might include some mechanism for performing a physical reorganization of the data in
a heap, which will get rid of forward pointers. Note that forward pointers exist only in heaps, and that DBCC
INDEXDEFRAG will not work on heap tables. You can defragment a nonclustered index on a heap but not the table
itself. Currently, when a forward pointer is created, it stays there forever—with only two exceptions. The first
exception is the case I already mentioned, in which a row shrinks enough to return to its original location. The second
exception is when the entire database shrinks. The bookmarks are actually reassigned when a file is shrunk. The
shrink process never generates forwarding pointers. For pages that were removed because of the shrinking process,
any forwarded rows or stubs they contain are effectively "unforwarded."

There are two ways to see the total count of forwarded records in a table. First, you can enable trace flag 2509 and
then execute the DBCC CHECKTABLE command:
DBCC TRACEON (2509)
GO

DBCC CHECKTABLE (bigrows)

You'll see results like this:

DBCC results for 'bigrows'.
There are 5 rows in 2 pages for object 'bigrows'.
Forwarded Record count = 1
This document is created with the unregistered version of CHM2PDF Pilot

DBCC execution completed. If DBCC printed error messages, contact your

system administrator.

The second way to see a count of forwarding pointers is to execute DBCC SHOWCONTIG with the
TABLERESULTS option. As I mentioned in Chapter 8, one of the extra columns returned with this option is the
number of forwarded records in the table.

Updating in Place
In SQL Server 2000, updating a row in place is the rule rather than the exception. This means that the row stays in
exactly the same location on the same page and only the bytes affected are changed. In addition, the log will contain a
single record for each such updated row unless the table has an update trigger on it or is marked for replication. In
these cases, the update still happens in place, but the log will contain a delete record followed by an insert record.

In cases where a row can't be updated in place, the cost of a not-in-place update is minimal because of the way the
nonclustered indexes are stored and because of the use of forwarding pointers. Updates will happen in place if a heap
is being updated or if a table with a clustered index is updated without any change to the clustering keys. You can also
get an update in place if the clustering key changes, but the row does not need to move. For example, if you have a
clustered index on a last name column containing consecutive key values of 'Able', 'Becker', and 'Charlie', you might
want to update 'Becker' to 'Baker'. Since the row would stay in the same location even after changing the clustered
index key, SQL Server will perform this as an update in place.

Updating Not in Place

If your update can't happen in place because you're updating clustering keys, it will occur as a delete followed by an
insert. In some cases, you'll get a hybrid update: Some of the rows will be updated in place and some won't. If you're
updating index keys, SQL Server builds a list of all the rows that need to change as both a delete and an insert
operation. This list is stored in memory, if it's small enough, and is written to tempdb if necessary. This list is then
sorted by key value and operator (delete or insert). If the index whose keys are changing isn't unique, the delete and
insert steps are then applied to the table. If the index is unique, an additional step is carried out to collapse delete and
insert operations on the same key into a single update operation.

Let's look at a simple example. The following code builds a table with a unique clustered index on column X and then
updates that column in both rows:
USE pubs
GO

DROP TABLE T1
GO

CREATE TABLE T1(X int PRIMARY KEY, Y int)

INSERT T1 VALUES(1, 10)
INSERT T1 VALUES(2, 20)
GO

UPDATE T1 SET X = 3 - X
GO

Table 9-3 shows the operations that SQL Server generates internally for the above update statement. This list of
operations is called the input stream and consists of both the old and new values of every column and an identifier for
each row that is to be changed.

Table 9-3. A SQL Server input stream for an update operation.

This document is created with the unregistered version of CHM2PDF Pilot

Operation Original X Original Y New X New Y Row ID

update 1 10 2 10 ROW1
update 2 20 1 20 ROW2

However, in this case the updates must be split into delete and insert operations. If they aren't, the update to the first
row, changing X from 1 to 2, will fail with a duplicate-key violation. A converted input stream is generated, as shown
in Table 9-4.

Table 9-4. Converted input stream for an update operation.

Operation Original X Original Y Row ID
delete 1 10 ROW1
insert 2 10 <null>
delete 2 20 ROW2
insert 1 20 <null>

Note that for insert operations, the Row ID isn't relevant in the input stream of operations to be performed. Before a
row is inserted, it has no Row ID. This input stream is then sorted by index key and operation, as in Table 9-5.

Table 9-5. Input stream sorted by index key and operation.

Operation X Y Row ID
delete 1 10 ROW1
insert 1 20 <null>
delete 2 20 ROW2
insert 2 10 <null>

Finally, if the same key value has both a delete and an insert, the two rows in the input stream are collapsed into an
update operation. The final input stream looks like Table 9-6.

Table 9-6. The final output stream for an update operation.

Operation Original X Original Y New X New Y Row ID
update 1 10 1 20 ROW1
update 2 20 2 10 ROW2

Note that, although the original query was an update to column X, after the split, sort, and collapse of the input stream
the final set of actions looks like we're updating column Y! This method of actually carrying out the update ensures that
no intermediate violations of the index's unique key occur.

Part of the graphical query plan for this update (see Figure 9-4) shows you the split, sort, and collapse phases. We'll
look at query plans and the mechanisms for examining query plans in Chapter 15.

Updates to any nonclustered index keys also affect the leaf level of the index by splitting the operation into deletes
and inserts. Think of the leaf level of a nonclustered index as a mini-clustered index, so any modification of the key
can potentially affect the sequence of values in the leaf level. Just like for the data in a clustered index, the actual type
of update is determined by whether the index is unique. If the nonclustered index is not unique, the update is split into
delete and insert operators. If the nonclustered index is unique, the split is followed by a sort and an attempt to
collapse any deletes and inserts on the same key back into an update operation.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 9-4. The update query plan displayed graphically.

Table-Level vs. Index-Level Data Modification

I've been discussing only the placement and index manipulation necessary for modifying either a single row or a few
rows with no more than a single index. If you are modifying multiple rows in a single insertion operation (insert,
update, or delete) or by using BCP or the BULK INSERT command and the table has multiple indexes, you must be
aware of some other issues. SQL Server 2000 offers two strategies for maintaining all the indexes that belong to a
table: table-level modification and index-level modification. The query optimizer chooses between them based on its
estimate of the anticipated execution costs for each strategy.

Table-level modification is sometimes called row at a time, and index-level modification is sometimes called index
at a time. In table-level modification, all indexes are maintained for each row as that row is modified. If the update
stream isn't sorted in any way, SQL Server has to do a lot of random index accesses, one access per index per
update row. If the update stream is sorted, it can't be sorted in more than one order, so there might be nonrandom
index accesses for at most one index.

In index-level modifications, SQL Server gathers all the rows to be modified and sorts them for each index. In other
words, there are as many sort operations as there are indexes. Then, for each index, the updates are merged into the
index, and each index page is never accessed more than once, even if multiple updates pertain to a single index leaf
page.

Clearly, if the update is small, say less than a handful of rows, and the table and its indexes are sizable, the query
optimizer usually considers table-level modification the best choice. Most OLTP operations use table-level
modification. On the other hand, if the update is relatively large, table-level modifications require a lot of random I/O
operations and might even read and write each leaf page in each index multiple times. In that case, index-level
modification offers much better performance. The amount of logging required is the same for both strategies.

Let's look at a specific example with some actual numbers. Suppose you use BULK INSERT to increase the size of
a table by 1 percent (which could correspond to one week in a two-year sales history table). The table is stored in a
heap (no clustered index), and the rows are simply appended at the end because there is not much available space on
other pages in the table. There's nothing to sort on for insertion into the heap, and the rows (and the required new
heap pages and extents) are simply appended at the end of the heap. Assume also that there are two nonclustered
indexes on columns a and b. The insert stream is sorted on column a and merged into the index on a.

If an index entry is 20 bytes long, an 8-KB page filled at 70 percent (which is the natural, self-stabilizing value in a
B-tree after lots of random insertions and deletions) would contain 8 KB * 70% / 20 bytes = 280 entries. Eight pages
(one extent) would then contain 2240 entries, assuming that the leaf pages are nicely laid out in extents. Table growth
This document is created with the unregistered version of CHM2PDF Pilot

of 1 percent would mean an average of 2.8 new entries per page, or 22.4 new entries per extent.

Table-level (row-at-a-time) insertion would touch each page an average of 2.8 times. Unless the buffer pool is large,
row-at-a-time insertion reads, updates, and writes each page 2.8 times. That's 5.6 I/O operations per page, or 44.8
I/O operations per extent. Index-level (index-at-a-time) insertion would read, update, and write each page (and
extent, again presuming a nice contiguous layout) exactly once. In the best case, about 45 page I/O operations are
replaced by 2 extent I/O operations for each extent. Of course, the cost of doing the insert this way also includes the
cost of sorting the inputs for each index. But the cost savings of the index-level insertion strategy can easily offset the
cost of sorting.
NOTE
Earlier versions of SQL Server recommended that you drop all your indexes if you were going to import a lot of
rows into a table using a bulkcopy interface because they offered no index-level strategy for maintaining all the
indexes.

You can determine whether your updates were done at the table level or the index level by inspecting the
SHOWPLAN output. (I'll discuss query plans and SHOWPLAN in more detail in Chapter 15.) If SQL Server
performs the update at the index level, you'll see a plan produced that contains an update operator for each of the
affected indexes. If SQL Server performs the update at the table level, you'll see only a single update operator in the
plan.

Logging
Standard INSERT, UPDATE, and DELETE statements are always logged to ensure atomicity, and you can't disable
logging of these operations. The modification must be known to be safely on disk in the transaction log (write-ahead
logging) before the commit of the statement or transaction can be acknowledged to the calling application. Page
allocations and deallocations, including those done by TRUNCATE TABLE, are also logged. Certain operations can
be minimally logged when your database is in the BULK_LOGGED recovery mode, but even then, information on
allocations and deallocations is written to the log, along with the fact that a minimally logged operation has been
executed.

Locking
Any data modification must always be protected with some form of exclusive lock. For the most part, SQL Server
makes all the locking decisions internally; a user or programmer doesn't need to request a particular kind of lock.
Chapter 14 explains the different types of locks and their compatibility. Chapter 15 discusses techniques for
overriding SQL Server's default locking behavior. However, because locking is closely tied to data modification, you
should always be aware of the following:
 Every type of data modification performed in SQL Server requires some form of exclusive lock. For most
data modification operations, SQL Server considers row locking as the default, but if many locks are
required, this could escalate to page locks or even a full table lock.
 Update locks can be used to signal the intention to do an update, and they are important for avoiding
deadlock conditions. But ultimately, the update operation requires that an exclusive lock be performed. The
update lock serializes access to ensure that an exclusive lock can be acquired, but the update lock isn't
sufficient by itself.
 Exclusive locks must always be held until the end of a transaction in case the transaction needs to be undone
(unlike shared locks, which can be released as soon as the scan moves off the page, assuming that READ
COMMITTED isolation is in effect).
 If a full table scan must be employed to find qualifying rows for an update or a delete, SQL Server has to
inspect every row to determine the row to modify. Other processes that need to find individual rows will be
blocked even if they will ultimately modify different rows. Without inspecting the row, SQL Server has no
way of knowing whether the row qualifies for the modification. If you're modifying only a subset of rows in the
table, as determined by a WHERE clause, be sure that you have indexes available to allow SQL Server to
This document is created with the unregistered version of CHM2PDF Pilot

access the needed rows directly so it doesn't have to scan every row in the table. (I'll talk about how to
choose the best indexes for your tables in Chapters 15 and 16.) Doing full table scans to modify rows won't
result in a high-concurrency environment.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
This chapter covered data modification within SQL Server. You modify most data using the three SQL DML
statements: INSERT, UPDATE, and DELETE. Although you can typically think of INSERT as affecting only one
row, SQL Server provides several forms of INSERT that can insert an entire result set at once. In addition, SQL
Server offers utilities and interfaces for fast bulk loading of data. UPDATE and DELETE operations can include
search operations to find the rows to modify, bringing to bear all the power of the SQL query language to specify the
rows.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 10

Programming with
Transact-SQL
Microsoft Transact-SQL allows you to use standard SQL as a programming language to write logic that can execute
within the database engine. This extremely powerful capability is one of the keys to the success and growth of
Microsoft SQL Server. Transact-SQL simplifies application development and reduces the amount of conversation
necessary between the client application and the server by allowing more code to be processed at the server.

In this book, I won't delve into all the available syntax of the Transact-SQL language. I'll leave that to the SQL
Server documentation. Instead, we'll see the significant programming constructs available, along with some comments
and examples. In Chapter 7, I examined the SELECT statement, which is the foundation for effective use of
Transact-SQL. Before reading any further, be sure you have a good understanding of the concepts presented in that
chapter.

This chapter focuses on the Transact-SQL programming extensions, such as control of flow, looping, error handling,
and environment-specific options. These extensions allow you to easily write complex routines entirely in SQL, and
they provide the basis for the additional power of Transact-SQL over standard SQL. Later in this chapter, I'll show
you some extensive examples that sometimes use Transact-SQL in nonintuitive ways, but it will be far from exhaustive
coverage of what is possible using the Transact-SQL language. In the last year, several books dealing specifically with
programming in SQL Server have been published, and another one is due out about the same time that this book is.
Those books are described in the bibliography, and I strongly suggest you take a look at one or more of them if you
want to see the full range of possibilities with SQL Server programming. In Chapter 11, we'll look at batches, stored
procedures, and functions. In Chapter 12, we'll look at transactions and triggers. Chapter 13 deals with cursors,
which bridge the set-based world of SQL with sequential processing. If you're interested in developing applications
using SQL Server, the information in all of these chapters will be important to you.
This document is created with the unregistered version of CHM2PDF Pilot

34

Transact-SQL as a Programming
Language
Is Transact-SQL really a programming language? On the one hand, Microsoft doesn't claim that Transact-SQL is an
alternative to C, C++, Microsoft Visual Basic, COBOL, Fortran, or such 4GL development environments as Sybase
PowerBuilder or Borland Inprise Delphi. You'd be hard-pressed to think of a substantial application that you could
write entirely in Transact-SQL, and every one of those other languages or tools exists to do exactly that. Also,
Transact-SQL offers no user interface and no file or device I/O, and the programming constructs are simple and
limited.

On the other hand, you can argue that Transact-SQL is a specialized language that is best used in addition to one of
those other languages or tools. It allows SQL Server to be programmed to execute complex tasks such as declaring
and setting variables, branching, looping, and error checking, without the need for code to be written in another
language. You can write reusable routines that you subsequently invoke and pass variables to. You can introduce
bugs if you make an error in logic or syntax. Your Transact-SQL code can quickly get so complicated that you'll be
glad you can use the same debugger to debug Transact-SQL routines that you use for C, C++, and Java
development.

With Transact-SQL routines, conditional logic executes within the SQL Server engine and even within an executing
SQL statement. This can greatly improve performance—the alternative would be message passing between the client
process and the server. In today's increasingly networked world, reducing round-trip conversations is a key to
developing and deploying efficient applications. Some might claim that the emergence of the Internet makes
client/server computing irrelevant. But in fact, the typically slow network conditions found in Internet applications
make client/server computing more important—operations should be written so that the client/server conversation is
minimized. The best way to do this is to use the programming aspects of Transact-SQL to let whole routines execute
remotely, without the need for intermediate processing at the client. The corresponding performance gains are
phenomenal.

In addition, by having SQL Server, rather than the application, handle the database logic, applications can often
better insulate themselves from change. An application can execute a procedure simply by calling it and passing some
parameters, and then the database structure or the procedure can change radically while the application remains
entirely unaffected. As long as the inputs and outputs of the procedure are unchanged, the application is shielded from
underlying changes to the database. The ability to encapsulate the database routines can result in a more efficient
development environment for large projects in which the application programmers are often distinct from the database
experts.

Programming at Multiple Levels

In the early days of database management systems, a single computer handled all processing. This was what we now
call the one-tier model. The advent of client/server computing introduced the classic two-tier model: a server
receiving requests from and providing data to a separate client. The newest approach, the three-tier model, places an
intermediate layer between the database server and the client application.

Figure 10-1 shows the two-tier client/server model, and Figure 10-2 shows the three-tier model, which some
programmers find preferable for some applications.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 10-1. The two-tier model.

Figure 10-2. The three-tier model.

The two-tier model has a client (first tier) and a server (second tier); the client handles application and presentation
logic, and the server handles data services and business services. The two-tier model uses the so-called fat client
—the client does a large amount of application processing locally. Many solutions are deployed using this topology;
certainly the lion's share of today's non-Internet client/server solutions are two-tier.

In the three-tier model, a thin client handles mostly presentation tasks. Supporters of the three-tier approach point
out that this model allows the client computer to be less powerful; allows a variety of client operating environments
(obviously the case with Internet browsers); and reduces the complexity of installing, configuring, and maintaining
software at the client. The client connects to an application server that handles the application process. The
application server then connects to the database server, handling data services. The application server is typically a
more powerful machine than the client, and it handles all the issues of application code maintenance, configuration,
operations, and so on.

Presumably, far fewer application servers exist than clients; therefore, server costs and issues should be reduced in
the three-tier model. (The three tiers are logical concepts; the actual number of computers involved might be more or
less than three.) Typically, you're part of a three-tier model when you use the Internet with a browser and access a
Web page residing on an Internet server that performs database work. An example of a three-tier application is a
mail-order company's Web page that checks current availability by using ODBC calls to access its inventory in SQL
Server.

Both two-tier and three-tier solutions have advantages and disadvantages, and each can be viewed as a potential
solution to a given problem. The specifics of the problem and the constraints imposed by the environment (including
costs, hardware, and expertise of both the development staff and the user community) should drive the solution's
overall design. However, some people claim that a three-tier solution means that such capabilities as stored
procedures, rules, constraints, or triggers are no longer important or shouldn't be used. In a three-tier solution, the
middle tier doing application logic is still a client from the perspective of the database services (that is, SQL Server).
Consequently, for that tier, it's still preferable that remote work be done close to the data. If the data can be accessed
This document is created with the unregistered version of CHM2PDF Pilot

by servers other than the application server, integrity checking must be done at the database server as well as in the
application. Otherwise, you've left a gaping hole through which the integrity checks in the application can be
circumvented.

Throughout the remainder of this book, unless otherwise noted, I'll use the term client from the perspective of SQL
Server. The client sends the commands and processes the results, and it makes no difference to SQL Server whether
an application server process does this or whether the process that the end user is running directly does it. The
benefits of the programmable server provide much more efficient network use and better abstraction of data services
from the application and application server.
A Three-Tier Solution That Works
Perhaps the single most successful three-tier solution available today is R/3 from SAP. The financial and operations
capabilities that R/3 provides use a three-tier model, with a terminal-like interface connected to the R/3 application
server that then accesses Microsoft SQL Server. The R/3 application server wisely and heavily uses stored
procedures that provide both performance gains and development efficiency.
This document is created with the unregistered version of CHM2PDF Pilot

34

Transact-SQL Programming
Constructs
Transact-SQL extends standard SQL by adding many useful programming constructs. These constructs will look
familiar to developers experienced with C/C++, Basic, Visual Basic, Fortran, Java, Pascal, and similar languages. The
Transact-SQL extensions are wonderful additions to the original SQL standard. Before the extensions were available,
requests to the database were always simple, single statements. Any conditional logic had to be provided by the
calling application. With a slow network or via the Internet, this would be disastrous, requiring many trips across the
network—more than are needed by simply letting conditions be evaluated at the server and allowing subsequent
branching of execution.

Variables
Without variables, a programming language wouldn't be particularly useful. Transact-SQL is no exception. Variables
that you declare are local variables, which have scope and visibility only within the batch or stored procedure in
which you declare them. (The next chapter discusses the details of batches and stored procedures.) In code, the
single @ character designates a local variable. SQL Server 2000 introduces the ability to store and retrieve session
context information, which allows information to be available throughout a connection; in any stored procedure,
function, or trigger; and across batches. You can use this session context information like a session variable, with a
scope broader than that of SQL Server's local variables but not as broad as that of a true global variable.

Transact-SQL has no true global variables that allow information to be shared across connections. Older
documentation refers to certain parameterless system functions as global variables because they're designated by
@@, which is similar to the designation for local variables. We'll look at these in more detail in Chapter 12, in the
section on system functions.

User-declared global variables would be a nice enhancement that we might see in a future release. Until then,
temporary tables provide a decent alternative for sharing values among connections.

Local Variables
You declare local variables at the beginning of a batch or a stored procedure. You can subsequently assign values to
local variables with the SELECT statement or the SET statement, using an equal (=) operator. The values you assign
using the SET statement can be constants, other variables, or expressions. When you assign values to a variable using
a SELECT statement, you typically select the values from a column in a table. The syntax for declaring and using the
variables is identical, regardless of the method you use to assign the values to the variable. In the following example, I
declare a variable, assign it a value using the SET statement, and then use that variable in a WHERE clause:
DECLARE @limit money
SET @limit = $10
SELECT * FROM titles
WHERE price <= @limit

I could have written the same code using only the SELECT statement instead of the SET statement. However, you
typically use SELECT when the values to be assigned are found in a column of a table. A SELECT statement used to
assign values to one or more variables is called an assignment SELECT. You can't combine the functionality of the
assignment SELECT and a "regular" SELECT in the same statement. That is, if a SELECT statement is used to assign
This document is created with the unregistered version of CHM2PDF Pilot

values to variables, it can't also return values to the client as a result set. In the following simple example, I declare two
variables, assign values to them from the roysched table, and then select their values as a result set:
DECLARE @min_range int, @hi_range int -- Variables declared
SELECT @min_range=MIN(lorange),
@hi_range=MAX(hirange) FROM roysched -- Variables assigned
SELECT @min_range, @hi_range -- Values of variables
-- returned as result

Note that a single DECLARE statement can declare multiple variables. When you use a SELECT statement for
assigning values, you can assign more than one value at a time. When you use SET to assign values to variables, you
must use a separate SET statement for each variable. For example, the following SET statement returns an error:
SET @min_range = 0, @hi_range = 100

Be careful when you assign variables by selecting a value from the database—you want to ensure that the SELECT
statement will return only one row. It's perfectly legal to assign a value to a variable in a SELECT statement that will
return multiple rows, but the variable's value might not be what you expect: no error message is returned, and the
variable has the value of the last row returned.

For example, suppose that the following stor_name values exist in the stores table of the pubs database:
stor_name
---------
Eric the Read Books
Barnum's
News & Brews
Doc-U-Mat: Quality Laundry and Books
Fricative Bookshop
Bookbeat

The following assignment runs without error but is probably a mistake:

DECLARE @stor_name varchar(30)
SELECT @stor_name=stor_name FROM stores

The resulting value of @stor_name is the last row, Bookbeat. But consider the order returned, without an ORDER
BY clause, as a chance occurrence. Assigning a variable to a SELECT statement that returns more than one row
usually isn't intended, and it's probably a bug introduced by the developer. To avoid this situation, you should qualify
the SELECT statement with an appropriate WHERE clause to limit the result set to the one row that meets your
criteria.

Alternatively, you can use an aggregate function, such as MAX, MIN, or SUM, to limit the number of rows returned.
Then you can select only the value you want. If you want the assignment to be to only one row of a SELECT
statement that might return many rows (and it's not important which row that is), you should at least use SELECT
TOP 1 to avoid the effort of gathering many rows, with all but one row thrown out. (In this case, the first row would
be returned, not the last. You could use ORDER BY to explicitly indicate which row should be first.) You might also
consider checking the value of @@ROWCOUNT immediately after the assignment and, if it's greater than 1, branch
off to an error routine.

Also be aware that if the SELECT statement assigning a value to a variable doesn't return any rows, nothing will be
assigned. The variable will keep whatever value it had before the assignment SELECT statement was run. For
example, suppose we use the same variable twice to find the first names of authors with a particular last name:
DECLARE @firstname varchar(20)
This document is created with the unregistered version of CHM2PDF Pilot

SELECT @firstname = au_fname

FROM authors
WHERE au_lname = 'Greene'
SELECT @firstname -- Return the first name as a result
SELECT @firstname = au_fname
FROM authors
WHERE au_lname = 'Ben-Gan'
SELECT @firstname -- Return the first name as a result

If you run the previous code, you'll see the same first name returned both times because no authors with the last name
of Ben-Gan exist, at least not in the pubs database!

You can also assign a value to a variable in an UPDATE statement. This approach can be useful and more efficient
than using separate statements to store the old value of a column and assign it a new value. The following UPDATE
will reduce the price of one book by 20 percent, and save the original price in a variable:
DECLARE @old_price money

UPDATE title
SET @old_price = price = price * 0.8
WHERE title_id = 'PC2091'

A variable can be assigned a value from a subquery, and in some cases it might look like there's no difference
between using a subquery and a straight SELECT. For example, the following two batches return exactly the same
result set:
-- First batch using a straight SELECT:
DECLARE @firstname varchar(20)
SELECT @firstname = au_fname
FROM authors
WHERE au_lname = 'Greene'
SELECT @firstname

-- Second batch using a subquery:

DECLARE @firstname varchar(20)
SELECT @firstname = ( SELECT au_fname
FROM authors
WHERE au_lname = 'Greene')
SELECT @firstname

However, if we leave off the WHERE clause, these two batches will behave very differently. As we saw earlier in the
example selecting from the stores table, the first batch will leave the variable with the value that the last row supplied:
DECLARE @firstname varchar(20)
SELECT @firstname = au_fname
FROM authors
SELECT @firstname

RESULT:

Akiko

This one row is returned because the assignment to the variable is happening for every row that the query would
return. With no WHERE clause, every row in the table is returned, and for every row, the first name value is assigned
to the variable, overriding whatever value was assigned by the previous row returned. We are left with the value from
the last row because there are no more values to override that one.
This document is created with the unregistered version of CHM2PDF Pilot

However, when I use a subquery and leave off the WHERE clause, the result is very different:
DECLARE @firstname varchar(20)
SELECT @firstname = (SELECT au_fname
FROM authors)
SELECT @firstname

RESULT:
Server: Msg 512, Level 16, State 1, Line 0
Subquery returned more than 1 value. This is not permitted when the
subquery follows =, !=, <, <= , >, >= or when the subquery is used
as an expression.

--------------------
NULL

In this case, the query returns an error. When you use a subquery, the entire subquery is processed before any
assignment to the variable is made. The subquery's result set contains 23 rows, and then all 23 rows are used in the
assignment to the variable. Since we can assign only a single value to a variable, the error message is generated.

Session Variables
SQL Server 2000 allows you to store and retrieve session-level context information by directly querying the
sysprocesses table in the master database. As I mentioned in Chapter 6, normally it is recommended that you never
access the system tables directly, but for this release, direct access of sysprocesses is the only way to retrieve this
information. A construct is supplied for setting this information, but you have nothing other than direct system table
access for retrieving it.

The sysprocesses table in the master database in SQL Server 2000 has a 128-byte field called context_info of
type binary, which means you have to manipulate the data using hexadecimal values. You can use the entire column
to store a single value or, if you're comfortable with manipulating binary values, you can use different groups of bytes
for different purposes. The following statements store the price of one particular book in the context_info column:
DECLARE @cost money
SELECT @cost = price FROM titles
WHERE title_id = 'bu1032'
SET context_info @cost

If we stored this value in a local variable, it would have the scope of only a single query batch and we wouldn't be
able to come back after executing several more batches and check the value. But storing it in the sysprocesses table
means that the value will be available as long as my connection is active. The sysprocesses table has one row for each
active process in SQL Server, and each row is assigned a unique Server Process ID (spid) value. A system function
@@spid returns the process ID for the current connection. Using @@spid, I can find the row in sysprocesses that
contains my context_info value.

Using SET context_info, I assigned a money value to the binary column. If you check the documentation for the
CONVERT function, you'll see that money is implicitly convertible to binary. For convenience, in Figure 10-3 I have
reproduced the chart that shows which datatypes are implicitly and explicitly convertible.

If, instead, I had tried to assign a character value to CONTEXT_INFO, I would have received an error message
because the chart in Figure 10-3 tells me that conversion from character to binary must be done explicitly. The SET
CONTEXT_INFO statement stores the assigned value in the fewest number of bytes, starting with the first byte. So I
can use the SUBSTRING function to extract a value from the context_info column, and since the money datatype is
stored in 8 bytes, I need to get the first 8 bytes of the column. I can then convert those first 8 bytes to money:
SELECT convert(money, substring(context_info, 1, 8))
This document is created with the unregistered version of CHM2PDF Pilot

FROM master..sysprocesses
WHERE spid = @@spid

Note that if I hadn't taken only the first 8 bytes of the context_info field, as shown below, and tried to convert that
directly to money, SQL Server would have assumed that the money value was in the last 8 bytes of the 128-byte
field, and return a 0.
SELECT convert(money, context_info)
FROM master..sysprocesses
WHERE spid = @@spid

Figure 10-3. Conversion table from the SQL Server documentation.

Since the context_info column is 128 bytes long, there's lot of room to work with. I can store another price in the
second set of 8 bytes if I'm careful. I need to take the first 8 bytes and concatenate them to the second price value
after converting it to hexadecimal. Concatenation works for hexadecimal values the same way as it does for character
strings, and the operator is the plus (+):
DECLARE @cost money, @context binary(128)
SELECT @cost = price FROM titles
WHERE title_id = 'ps2091'
SELECT @context =
convert(binary(8),context_info) + convert(binary(8), @cost)
FROM master..sysprocesses
WHERE spid = @@spid
SET context_info @context

Now the second 8 bytes contains the second price, and if I use the SUBSTRING function appropriately, I can
extract either price value from the context_info column:
This document is created with the unregistered version of CHM2PDF Pilot

SELECT First_price = convert(money, substring(context_info, 1, 8))

FROM master..sysprocesses
WHERE spid = @@spid

SELECT Second_price = convert(money, substring(context_info, 9, 8))

FROM master..sysprocesses
WHERE spid = @@spid

Control-of-Flow Tools
Like any programming language worth its salt, Transact-SQL provides a decent set of control-of-flow tools. (True,
Transact-SQL has only a handful of tools—perhaps not as many as you'd like—but they're enough to make it
dramatically more powerful than standard SQL.) The control-of-flow tools include conditional logic (IF?ELSE and
CASE), loops (only WHILE, but it comes with CONTINUE and BREAK options), unconditional branching
(GOTO), and the ability to return a status value to a calling routine (RETURN). Table 10-1 presents a quick
summary of these control-of-flow constructs, which you'll see used at various times in this book.

Table 10-1. Control-of-flow constructs in Transact-SQL.

Construct Description
Defines a statement block. Using BEGIN?END allows a
group of statements to be executed. Typically, BEGIN
immediately follows IF, ELSE, or WHILE. (Otherwise,
BEGIN?END
only the next statement will be executed.) For C
programmers, BEGIN?END is similar to using a {..}
block.
Continues processing at the statement following the
GOTO label
specified label.
Defines conditional and, optionally, alternate execution
IF?ELSE
when a condition is false.
Exits unconditionally. Typically used in a stored procedure
or trigger (although it can also be used in a batch).
RETURN [n] Optionally, a whole number n (positive or negative) can
be set as the return status, which can be assigned to a
variable when you execute the stored procedure.
Sets a time for statement execution. The time can be a
WAITFOR delay interval (up to 24 hours) or a specific time of day.
The time can be supplied as a literal or with a variable.
The basic looping construct for SQL Server. Repeats a
WHILE
statement (or block) while a specific condition is true.
?BREAK Exits the innermost WHILE loop.
?CONTINUE Restarts a WHILE loop.

CASE
The CASE expression is enormously powerful. Although CASE is part of the ANSI SQL-92 specification, it's not
required for ANSI compliance certification, and few database products other than Microsoft SQL Server have
implemented it. If you have experience with an SQL database system other than Microsoft SQL Server, chances are
you haven't used CASE. If that's the case (pun intended), you should get familiar with it now. It will be time well
spent. CASE was added in version 6, and once you get used to using it, you'll wonder how SQL programmers ever
managed without it.
This document is created with the unregistered version of CHM2PDF Pilot

CASE is a conceptually simpler way to do IF-ELSE IF-ELSE IF-ELSE IF-ELSE_type operations. It's roughly
equivalent to a switch statement in C. However, CASE is far more than shorthand for IF—in fact, it's not really even
that. CASE is an expression, not a control-of-flow keyword, which means that it can be used only inside other
statements. You can use CASE in a SELECT statement in the SELECT list, in a GROUP BY clause, or in an
ORDER BY clause. You can use it in the SET clause of an UPDATE statement. You can include CASE in the values
list for an INSERT statement. You can also use CASE in a WHERE clause in any SELECT, UPDATE, or DELETE
statement. Anywhere that Transact-SQL expects an expression, you can use CASE to determine the value of that
expression.

Here's a simple example of CASE. Suppose we want to classify books in the pubs database by price. We want to
segregate them as low-priced, moderately priced, or expensive. And we'd like to do this with a single SELECT
statement and not use UNION. Without CASE, we can't do it. With CASE, it's a snap:
SELECT
title,
price,
'classification'=CASE
WHEN price < 10.00 THEN 'Low Priced'
WHEN price BETWEEN 10.00 AND 20.00 THEN 'Moderately Priced'
WHEN price > 20.00 THEN 'Expensive'
ELSE 'Unknown'
END
FROM titles

Notice that even in this example, we have to worry about NULL because a NULL price won't fit into any of the
category buckets.

Here are the abbreviated results.

title price classification
------------------------------------- ------ -----------------
The Busy Executive's Database Guide 19.99 Moderately Priced
Cooking with Computers: Surreptitious 11.95 Moderately Priced
Balance Sheets
You Can Combat Computer Stress! 2.99 Low Priced
Straight Talk About Computers 19.99 Moderately Priced
Silicon Valley Gastronomic Treats 19.99 Moderately Priced
The Gourmet Microwave 2.99 Low Priced
The Psychology of Computer Cooking NULL Unknown
But Is It User Friendly? 22.95 Expensive

If all the conditions after your WHEN clauses are equality comparisons against the same value, you can use an even
simpler form so that you don't have to keep repeating the expression. Suppose we want to print the type of each
book in the titles table in a slightly more user-friendly fashion—for example, Modern Cooking instead of mod_cook.
We could use the following SELECT statement:
SELECT
title,
price,
'Type' = CASE
WHEN type = 'mod_cook' THEN 'Modern Cooking'
WHEN type = 'trad_cook' THEN 'Traditional Cooking'
WHEN type = 'psychology' THEN 'Psychology'
WHEN type = 'business' THEN 'Business'
WHEN type = 'popular_comp' THEN 'Popular Computing'
ELSE 'Not yet decided'
END
FROM titles
This document is created with the unregistered version of CHM2PDF Pilot

In this example, because every condition after WHEN is testing for equality with the value in the type column, we can
use an even simpler form. You can think of this simplification as factoring out the type = from every WHEN clause,
and you can place the type column at the top of the CASE expression:
SELECT
title,
price,
'Type' = CASE type
WHEN 'mod_cook' THEN 'Modern Cooking'
WHEN 'trad_cook' THEN 'Traditional Cooking'
WHEN 'psychology' THEN 'Psychology'
WHEN 'business' THEN 'Business'
ELSE 'Not yet decided'
END
FROM titles

You can use CASE in some unusual places; because it's an expression, you can use it anywhere that an expression is
legal. Using CASE in a view is a nice technique that can make your database more usable to others. For example, in
Chapter 7, we saw CASE in a view using CUBE to shield users from the complexity of "grouping NULL" values.
Using CASE in an UPDATE statement can make the update easier. More important, CASE can allow you to make
changes in a single pass of the data that would otherwise require you to do multiple UPDATE statements, each of
which would have to scan the data.

Cousins of CASE
SQL Server provides three nice shorthand derivatives of CASE: COALESCE, NULLIF, and ISNULL.
COALESCE and NULLIF are both part of the ANSI SQL-92 specification; they were added to version 6 at the
same time that CASE was added. ISNULL is a longtime SQL Server function.

COALESCE This function is equivalent to a CASE expression that returns the first NOT NULL expression in a list
of expressions.

COALESCE(expression1, expression2, ... expressionN)

If no non-null values are found, COALESCE returns NULL (which is what expressionN was, given that there were
no non-null values). Written as a CASE expression, COALESCE looks like this:
CASE
WHEN expression1 IS NOT NULL THEN expression1
WHEN expression2 IS NOT NULL THEN expression2

ELSE expressionN
END

NULLIF This function is equivalent to a CASE expression in which NULL is returned if expression1 =
expression2.

NULLIF(expression1, expression2)

If the expressions aren't equal, expression1 is returned. In Chapter 7, you saw that NULLIF can be handy if you use
dummy values instead of NULL, but you don't want those dummy values to skew the results produced by aggregate
functions. Written using CASE, NULLIF looks like this:
This document is created with the unregistered version of CHM2PDF Pilot

CASE
WHEN expression1=expression2 THEN NULL
ELSE expression1
END

ISNULL This function is almost the mirror image of NULLIF:

ISNULL(expression1, expression2)

ISNULL allows you to easily substitute an alternative expression or value for an expression that's NULL. In Chapter
7, you saw the use of ISNULL to substitute the string 'ALL' for a GROUPING NULL when using CUBE and to
substitute a string of question marks ('????') for a NULL value. The results were more clear and intuitive to users.
The functionality of ISNULL written instead using CASE looks like this:
CASE
WHEN expression1 IS NULL THEN expression2
ELSE expression1
END

The second argument to ISNULL (the value to be returned if the first argument is NULL) can be an expression or
even a SELECT statement. Say, for example, that we want to query the titles table. If a title has NULL for price, we
want to instead use the lowest price that exists for any book. Without CASE or ISNULL, this isn't an easy query to
write. With ISNULL, it is easy:
SELECT title, pub_id, ISNULL(price, (SELECT MIN(price)
FROM titles)) FROM titles

For comparison, here's an equivalent SELECT statement written with the longhand CASE formulation:
SELECT title, pub_id,
CASE WHEN price IS NULL THEN (SELECT MIN(price) FROM titles)
ELSE price
END
FROM titles

PRINT
Transact-SQL, like other programming languages, provides printing capability through the PRINT and RAISERROR
statements. I'll discuss RAISERROR momentarily, but first I'll take a look at the PRINT statement.

PRINT is the most basic way to display any character string of up to 8000 characters. You can display a literal
character string or a variable of type char, varchar, nchar, nvarchar, datetime, or smalldatetime. You can
concatenate strings and use functions that return string or date values. In addition, PRINT can print any expression or
function that is implicitly convertible to a character datatype. (Refer to Figure 10-3 to determine which datatypes are
implicitly convertible.)

It seems like every time you learn a new programming language, the first assignment is to write the "Hello World"
program. Using Transact-SQL, it couldn't be easier:
PRINT 'Hello World'

Even before you saw PRINT, you could've produced the same output in your SQL Query Analyzer screen with the
This document is created with the unregistered version of CHM2PDF Pilot

following:
SELECT 'Hello World'

So what's the difference between the two? The PRINT statement returns a message, nothing more. The SELECT
statement returns output like this:
-----------
Hello World

(1 row(s) affected)

The SELECT statement returns the string as a result set. That's why we get the (1 row(s) affected) message. When
you use SQL Query Analyzer to submit your queries and view your results, you might not see much difference
between the PRINT and the SELECT statements. But for other client tools (besides ISQL, OSQL, and SQL Query
Analyzer), there could be a big difference. PRINT returns a message of severity 0, and you need to check the
documentation for the client API that you're using to determine how messages are handled. For example, if you're
writing an ODBC program, the results of a PRINT statement are handled by SQLError.

On the other hand, a SELECT statement returns a set of data, just like selecting prices from a table does. Your client
interface might expect you to bind data to local (client-side) variables before you can use them in your client
programs. Again, you'll have to check your documentation to know for sure. The important distinction to remember
here is that PRINT returns a message and SELECT returns data.

PRINT is extremely simple, both in its use and in its capabilities. It's not nearly as powerful as, say, printf in C. You
can't give positional parameters or control formatting. For example, you can't do this:
PRINT 'Today is %s', GETDATE()

Part of the problem here is that %s is meaningless. In addition, the built-in function GETDATE returns a value of
type datetime, which can't be concatenated to the string constant. So, you can do this instead:
PRINT 'Today is ' + CONVERT(char(30), GETDATE())

RAISERROR
RAISERROR is similar to PRINT but can be much more powerful. RAISERROR also sends a message (not a result
set) to the client; however, RAISERROR allows you to specify the specific error number and severity of the message.
It also allows you to reference an error number, the text of which you can add to the sysmessages table.
RAISERROR makes it easy for you to develop international versions of your software. Rather than using multiple
versions of your SQL code when you want to change the language of the text, you need only one version. You can
simply add the text of the message in multiple languages. The user will see the message in the language used by the
particular connection.

Unlike PRINT, RAISERROR accepts printf-like parameter substitution as well as formatting control. RAISERROR
can also allow messages to be logged to the Windows NT or Windows 2000 event service, making the messages
visible to the Windows NT or Windows 2000 Event Viewer, and it allows SQL Server Agent alerts to be configured
for these events. In many cases, RAISERROR is preferable to PRINT, and you might choose to use PRINT only for
quick-and-dirty situations. For production-caliber code, RAISERROR is usually a better choice. Here's the syntax:

RAISERROR ({msg_id | msg_str}, severity, state[, argument1

[, argumentn]])
[WITH LOG]|[WITH NOWAIT]
This document is created with the unregistered version of CHM2PDF Pilot

After you call RAISERROR, the system function @@ERROR returns the value passed as msg_id. If no ID is
passed, the error is raised with error number 50000, and @@ERROR is set to that number. Error numbers lower
than 50000 are reserved for SQL Server use, so choose a higher number for your own messages.

You can also set the severity level; an informational message should be considered severity 0 (or severity 10; 0 and
10 are used interchangeably for a historical reason that's not important). Only someone with the system administrator
(SA) role can raise an error with a severity of 19 or higher. Errors of severity 20 and higher are considered fatal, and
the connection to the client is automatically terminated.

By default, if a batch includes a WAITFOR statement, results or messages are not returned to the client until the
WAITFOR has completed. If you want the messages generated by your RAISERROR statement to be returned to
the client immediately, even if the batch contains a subsequent WAITFOR, you can use the WITH NOWAIT option

You must use the WITH LOG option for all errors of severity 19 or higher. For errors of severity 20 or higher, the
message text isn't returned to the client. Instead, it's sent to the SQL Server error log, and the error number, text,
severity, and state are written to the operating system event log. A SQL Server system administrator can use the
WITH LOG option with any error number or any severity level.

Errors written to the operating system event log from RAISERROR go to the application event log, with
MSSQLServer as the source if the server is the default server, and MSSQL$InstanceName as the source if your
server is a named instance. These errors have event ID 17060. (Errors that SQL Server raises can have different
event IDs, depending on which component they come from.) The type of message in the event log depends on the
severity used in the RAISERROR statement. Messages with severity 14 and lower are recorded as informational
messages, severity 15 messages are warnings, and severity 16 and higher messages are errors. Note that the
severity used by the event log is the number that's passed to RAISERROR, regardless of the stated severity in
sysmessages. So, although I don't recommend it, you can raise a serious message as informational or a noncritical
message as an error. If you look at the data section of the event log for a message raised from RAISERROR, or if
you're reading from the event log in a program, you can choose to display the data as bytes or words. I find that when
I display the error data as words, it's easy to see that the first word is the error number, and the second word is the
severity (in hexadecimal, of course!).

You can use any number from 1 through 127 as the state parameter. You might pass a line number that tells where
the error was raised. But in a user-defined error message, state has no real relevance to SQL Server. For internally
generated error messages, the value of state is an indication of where in the code the error came from. It can
sometimes be of use to Microsoft Product Support personnel to know the value of state that was returned with an
error message.
TIP
There's a little-known way to cause isql.exe or OSQL.EXE (the command-line query tools) to terminate. Raising any
error with state 127 causes isql or OSQL to immediately exit, with a return code equal to the error number. (You can
determine the error number from a batch command file by inspecting the system ERRORLEVEL value.) You could
write your application so that it does something similar—it's a simple change to the error-handling logic. This trick can
be a useful way to terminate a set of scripts that would be run through isql or OSQL. You could get a similar result by
raising a high-severity error to terminate the connection. But using a state of 127 is actually simpler, and no scary
message will be written to the error log or event log for what is a planned, routine action.

You can add your own messages and text for a message in multiple languages by using the sp_addmessage stored
procedure. By default, SQL Server uses U.S. English, but you can add languages. SQL Server distinguishes U.S.
English from British English because of locale settings, such as currency. The text of messages is the same. The
procedure sp_helplanguage shows you what languages are available on your SQL Server instance. If a language you
are interested in doesn't appear in the list returned by sp_helplanguage, take a look at the script instlang.sql, which
can be found in the INSTALL subdirectory of your SQL Server instance installation files. This script adds locale
settings for additional languages but does not add the actual translated error messages. Suppose we want to rewrite
This document is created with the unregistered version of CHM2PDF Pilot

"Hello World" to use RAISERROR, and we want the text of the message to be able to appear in both U.S. English
and German (language ID of 1). Here's how:
EXEC sp_addmessage 50001, 10, 'Hello World', @replace='REPLACE'
-- New message 50001 for U.S. English
EXEC sp_addmessage 50001, 10, 'Hallo Welt' , @lang='Deutsch',
@replace='REPLACE'
-- New message 50001 for German

When RAISERROR is executed, the text of the message becomes dependent on the SET LANGUAGE setting of
the connection. If the connection does a SET LANGUAGE Deutsch, the text returned for RAISERROR (50001, 10,
1) will be Hallo Welt. If the text for a language doesn't exist, the U.S. English text (by default) is returned. For this
reason, the U.S. English message must be added before the text for another language is added. If no entry exists, even
in U.S. English, error 2758 results:
Msg 2758, Level 16, State 1
RAISERROR could not locate entry for error n in Sysmessages.

We could easily enhance the message to say whom the "Hello" is from by providing a parameter marker and passing
it when calling RAISERROR. (For illustration, we'll use some printf-style formatting, #6x, to display the process
number in hexadecimal format.)
EXEC sp_addmessage 50001, 10,
'Hello World, from: %s, process id: %#6x', @replace='REPLACE'

When user kalend executes

DECLARE @parm1 varchar(30), @parm2 int
SELECT @parm1=USER_NAME(), @parm2=@@spid
RAISERROR (50001, 15, -1, @parm1, @parm2)

this error is raised:

Msg 50001, Level 15, State 1
Hello, World, from: kalend, process id: 0xc

FORMATMESSAGE
If you have created your own error messages with parameter markers, you might need to inspect the entire message
with the parameters replaced by actual values. RAISERROR makes the message string available to the client but not
to your SQL Server application. To construct a full message string from a parameterized message in the sysmessages
table, you can use the function FORMATMESSAGE.

If we consider the last example from the preceding RAISERROR section, we can use FORMATMESSAGE to
build the entire message string and save it in a local variable for further processing. Here's an example of the use of
FORMATMESSAGE:
DECLARE @parm1 varchar(30), @parm2 int, @message varchar(100)
SELECT @parm1=USER_NAME(), @parm2=@@spid
SELECT @message = FORMATMESSAGE(50001, @parm1, @parm2)
PRINT 'The message is: ' + @message
This document is created with the unregistered version of CHM2PDF Pilot

This returns the following:

The message is: Hello World, from: kalend, process id: Oxc

Note that no error number or state information is returned.

Operators
Transact-SQL provides a large collection of operators for doing arithmetic, comparing values, doing bit operations,
and concatenating strings. These operators are similar to those you might be familiar with in other programming
languages. You can use Transact-SQL operators in any expression, including in the select list of a query, in the
WHERE or HAVING clauses of queries, in UPDATE and IF statements, and in CASE expressions.

Arithmetic Operators
We won't go into much detail here because arithmetic operators are pretty standard stuff. If you need more
information, please refer to the online documentation. Table 10-2 shows the arithmetic operators.

Table 10-2. Arithmetic operators in Transact-SQL.

Symbol Operation Used with These Datatypes
int, smallint, tinyint, bigint, numeric
, decimal, float, real, money,
+ Addition
smallmoney, datetime and
smalldatetime
int, smallint, tinyint, bigint,
numeric, decimal, float, real,
- Subtraction
money, smallmoney, datetime and
smalldatetime
int, smallint, tinyint, bigint,
* Multiplication numeric, decimal, float, real,
money, and smallmoney
int, smallint, tinyint, bigint,
/ Division numeric, decimal, float, real,
money, and smallmoney
% Modulo int, smallint, tinyint, and bigint

As with other programming languages, in Transact-SQL you need to consider the datatypes you're using when you
perform arithmetic operations. Otherwise, you might not get the results you expect. For example, which of the
following is correct?
1 / 2 = 0
1 / 2 = 0.5

If the underlying datatypes are both integers (including bigint, tinyint and smallint), the correct answer is 0. If one
or both of the datatypes are float (including real) or numeric/decimal, the correct answer is 0.5 (or 0.50000,
depending on the precision and scale of the datatype). When an operator combines two expressions of different
datatypes, the datatype precedence rules specify which datatype is converted to the other. The datatype with the
lower precedence is converted to the datatype with the higher precedence.

Here's the precedence order for SQL Server datatypes:

This document is created with the unregistered version of CHM2PDF Pilot

sql_variant (highest)
datetime
smalldatetime
float
real
decimal
money
smallmoney
bigint
int
smallint
tinyint
bit
ntext
text
image
timestamp (rowversion)
uniqueidentifier
nvarchar
nchar
varchar
char
varbinary
binary (lowest)

As you can see from the precedence list, an int multiplied by a float produces a float. If you need to use data of
type sql_variant in an arithmetic operation, you must first convert the sql_variant data to its base datatype. You can
explicitly convert to a given datatype by using the CAST function or its predecessor, CONVERT. I'll discuss these
conversion functions later in this chapter.

The addition (+) operator is also used to concatenate character strings. In this example, we want to concatenate last
names, followed by a comma and a space, followed by first names from the authors table (in pubs), and then we
want to return the result as a single column:
SELECT 'author'=au_lname + ', ' + au_fname FROM authors

Here's the abbreviated result:

author
------
White, Johnson
Green, Marjorie
Carson, Cheryl
O'Leary, Michael

Bit Operators
Table 10-3 shows the SQL Server bit operators.

Table 10-3. SQL Server bit operators.

Symbol Meaning
& Bitwise AND (two operands)
| Bitwise OR (two operands)
^ Bitwise exclusive OR (two operands)
~ Bitwise NOT (one operand)
This document is created with the unregistered version of CHM2PDF Pilot

The operands for the two-operand bitwise operators can be any of the datatypes of the integer or binary string
datatype categories (except for the image datatype), with the exception that both operands cannot be a type of
binary string. So, one of the operands can be binary or varbinary, but they can't both be binary or varbinary. In
addition, the right operand can be of the bit datatype only if the left operand is of type bit. Table 10-4 shows the
supported operand datatypes.

Table 10-4. Datatypes supported for two-operand bitwise operators.

Left Operand Right Operand
binary int, smallint, tinyint or bigint
bit int, smallint, tinyint, bigint or bit
int int, smallint, tinyint, bigint, binary, or varbinary
smallint int, smallint, tinyint, bigint, binary, or varbinary
tinyint int, smallint, tinyint, bigint binary, or varbinary
bigint int, smallint, tinyint, bigint, binary, or varbinary
varbinary int, smallint, tinyint, or bigint

The single operand for the bitwise NOT operator must be one of the integer datatypes.

Often, you must keep a lot of indicator-type values in a database, and bit operators make it easy to set up bit masks
with a single column to do this. The SQL Server system tables use bit masks. For example, the status field in the
sysdatabases table is a bit mask. If we wanted to see all databases marked as read-only, which is the 11th bit or
decimal 1024 (210), we could use this query:
SELECT 'read only databases'=name FROM master..sysdatabases
WHERE status & 1024 > 0

This example is for illustrative purposes only. As I mentioned in Chapter 6, in general, you shouldn't query the system
catalogs directly. Sometimes the catalogs need to change between product releases, and if you query directly, your
applications can break because of these required changes. Instead, you should use the provided system catalog
stored procedures and object property functions, which return catalog information in a standard way. If the underlying
catalogs change in subsequent releases, the property functions are also updated, insulating your application from
unexpected changes.

So, for this example, to see whether a particular database was marked read-only, we could use the
DATABASEPROPERTY function:
SELECT DATABASEPROPERTY('mydb', 'IsReadOnly')

A return value of 1 would mean that mydb was set to read-only status.

Note that keeping such indicators as bit datatype columns can be a better approach than using an int as a bit mask.
It's more straightforward—you don't always need to look up your bit-mask values for each indicator. To write the
"read only databases" query we just looked at, you'd need to check the documentation for the bit-mask value for
"read only." And many developers, not to mention end users, aren't that comfortable using bit operators and are likely
to use them incorrectly. From a storage perspective, bit columns don't require more space than equivalent bit-mask
columns require. (Eight bit fields can share the same byte of storage.)

But you wouldn't typically want to do that with a "yes/no" indicator field anyway. If you have a huge number of
indicator fields, using bit-mask columns of integers instead of bit columns might be a better alternative than dealing
with hundreds of individual bit columns. However, there's nothing to stop you from having hundreds of bit columns.
You'd have to have an awful lot of bit fields to actually exceed the column limit because a table in SQL Server can
This document is created with the unregistered version of CHM2PDF Pilot

have up to 1024 columns. If you frequently add new indicators, using a bit mask on an integer might be better. To add
a new indicator, you can make use of an unused bit of an integer status field rather than do an ALTER TABLE and
add a new column (which you must do if you use a bit column). The case for using a bit column boils down to clarity.

Comparison Operators
Table 10-5 shows the SQL Server comparison operators.

Table 10-5. SQL Server comparison operators.

Symbol Meaning
= Equal to
> Greater than
< Less than
>= Greater than or equal to
<= Less than or equal to
<> Not equal to (ANSI standard)
!= Not equal to (not ANSI standard)
!> Not greater than (not ANSI standard)
!< Not less than (not ANSI standard)

Comparison operators are straightforward. Only a few related issues might be confusing to a novice SQL Server
developer:
 When you deal with NULL, remember the issues with three-value logic and the truth tables that were
discussed in Chapter 7. Also understand that NULL is an unknown.
 Comparisons of char and varchar data (and their Unicode counterparts) depend on the collation value
associated with the column. Whether comparisons evaluate to TRUE depends on the sort order installed. You
can review the discussion of collation in Chapter 4.
 Comparison of sql_variant values involves special rules that break down the SQL Server datatype hierarchy
shown previously into datatype families, as shown in Table 10-6. The following script creates a table that I
can use to illustrate some of the issues involved in comparing values of type sql_variant:
CREATE TABLE variant2
(a sql_variant, b sql_variant )
GO
INSERT INTO variant2 VALUES (CAST (111 as int), CAST(222 as money ))
GO
INSERT INTO variant2 VALUES (CAST (333 as int), CAST(444 as char(3) ))
GO

Here are the special rules governing comparison of sql_variant values:

 When sql_variant values of different base datatypes are compared and the base datatypes are in different
datatype families, the value whose datatype family is higher in the hierarchy chart is considered the higher of
the two values.

You can see this behavior by executing the following query:

SELECT *
FROM variant2
WHERE a > b

Result:
a b
----- -----
333 444
This document is created with the unregistered version of CHM2PDF Pilot

The second row inserted had a value of base type int and a value of base type char. Even though you would
normally think that 333 is not greater than 444 because 333 is of datatype int and the family exact numeric,
it is considered higher than 444, of datatype char, and of the family Unicode.
 When sql_variant values of different base datatypes are compared and the base datatypes are in the same
datatype family, the value whose base datatype is lower in the hierarchy chart is implicitly converted to the
other datatype and the comparison is then made.

You can see this behavior by executing the following query:

SELECT *
FROM variant2
WHERE a < b

Result:
a b
-------------------- -----------------
222.0000

In this case, the two base types are int and money, which are in the same family, so 111 is considered less
than 222.0000.
 When sql_variant values of the char, varchar, nchar, or varchar datatypes are compared, they are
evaluated based on additional criteria, including the locale code ID (LCID), the LCID version, the
comparison flags use for the column's collation, and the sort ID.

Table 10-6. Datatype families for use when comparing sql_variant values.
Datatype Hierarchy Datatype Family
sql_variant sql_variant
datetime datetime
smalldatetime datetime
float approximate numeric
real approximate numeric
decimal exact numeric
money exact numeric
smallmoney exact numeric
bigint exact numeric
int exact numeric
smallint exact numeric
tinyint exact numeric
bit exact numeric
nvarchar Unicode
nchar Unicode
varchar Unicode
char Unicode
varbinary binary
binary binary
uniqueidentifier uniqueidentifier

Logical and Grouping Operators (Parentheses)

This document is created with the unregistered version of CHM2PDF Pilot

The three logical operators (AND, OR, and NOT) are vitally important in expressions, especially in the WHERE
clause. You can use parentheses to group expressions and then apply logical operations to the group. In many cases,
it's impossible to correctly formulate a query without using parentheses. In other cases, you might be able to avoid
them, but your expression won't be very intuitive and the absence of parentheses could result in bugs. You could
construct a convoluted expression using some combination of string concatenations or functions, bit operations, or
arithmetic operations instead of using parentheses, but there's no good reason to do this.

Using parentheses can make your code clearer, less prone to bugs, and easier to maintain. Because there's no
performance penalty beyond parsing, you should use parentheses liberally. Below is a query that you couldn't readily
write without using parentheses. This query is simple, and in English the request is well understood. But you're also
likely to get this query wrong in Transact-SQL if you're not careful, and it clearly illustrates just how easy it is to make
a mistake.
Find authors who do not live in either Salt Lake City, UT, or Oakland, CA.

Following are four examples that attempt to formulate this query. Three of them are wrong. Test yourself: which one
is correctly stated? (As usual, multiple correct formulations are possible—not just the one presented here.)

EXAMPLE 1
SELECT au_lname, au_fname, city, state FROM authors
WHERE
city <> 'OAKLAND' AND state <> 'CA'
OR
city <> 'Salt Lake City' AND state <> 'UT'

EXAMPLE 2
SELECT au_lname, au_fname, city, state FROM authors
WHERE
(city <> 'OAKLAND' AND state <> 'CA')
OR
(city <> 'Salt Lake City' AND state <> 'UT')

EXAMPLE 3
SELECT au_lname, au_fname, city, state FROM authors
WHERE
(city <> 'OAKLAND' AND state <> 'CA')
AND
(city <> 'Salt Lake City' AND state <> 'UT')

EXAMPLE 4
SELECT au_lname, au_fname, city, state FROM authors
WHERE
NOT
(
(city='OAKLAND' AND state='CA')
OR
(city='Salt Lake City' AND state='UT')
)

I hope you can see that only Example 4 operates correctly. This query would be much more difficult to write without
This document is created with the unregistered version of CHM2PDF Pilot

some combination of parentheses, NOT, OR (including IN, a shorthand for OR), and AND.

You can also use parentheses to change the order of precedence in mathematical and logical operations. These two
statements return different results:
SELECT 1.0 + 3.0 / 4.0 -- Returns 1.75
SELECT (1.0 + 3.0) / 4.0 -- Returns 1.00

The order of precedence is similar to what you learned back in algebra class (except for the bit operators).
Operators of the same level are evaluated left to right. You use parentheses to change precedence levels to suit your
needs, when you're unsure, or when you simply want to make your code more readable. Groupings with parentheses
are evaluated from the innermost grouping outward. Table 10-7 shows the order of precedence.

Table 10-7. Order of precedence in mathematical and logical operations, from highest to lowest.
Operation Operators
Bitwise NOT ~
Multiplication/division/modulo */%
Addition/subtraction +/-
Bitwise exclusive OR ^
Bitwise AND &
Bitwise OR |
Logical NOT NOT
Logical AND AND
Logical OR OR

If you didn't pick the correct example, you can easily see the flaw in Examples 1, 2, and 3 by examining the output.
Examples 1 and 2 both return every row. Every row is either not in CA or not in UT because a row can be in only
one or the other. A row in CA isn't in UT, so the expression returns TRUE. Example 3 is too restrictive—for
example, what about the rows in San Francisco, CA? The condition (city <> 'OAKLAND' AND state <> 'CA')
would evaluate to (TRUE and FALSE) for San Francisco, CA, which, of course, is FALSE, so the row would be
rejected when it should be selected, according to the desired semantics.

Scalar Functions
In Chapter 7, we looked at aggregate functions, such as MAX, SUM, and COUNT. These functions operate on a
set of values to produce a single aggregated value. In addition to aggregate functions, SQL Server provides scalar
functions, which operate on a single value. Scalar is just a fancy term for single value. You can also think of the
value in a single column of a single row as scalar. Scalar functions are enormously useful—the Swiss army knife of
SQL Server. You've probably noticed that scalar functions have been used in several examples already—we'd all be
lost without them.

You can use scalar functions in any legal expression, such as:
 In the select list
 In a WHERE clause, including one that defines a view
 Inside a CASE expression
 In a CHECK constraint
 In the VALUES clause of an INSERT statement
 In an UPDATE statement to the right of the equals sign in the SET clause
 With a variable assignment
 As a parameter inside another scalar function
This document is created with the unregistered version of CHM2PDF Pilot

SQL Server provides too many scalar functions to remember. The best you can do is to familiarize yourself with those
that exist, and then, when you encounter a problem and a light bulb goes on in your head to remind you of a function
that will help, you can look at the online documentation for details. Even with so many functions available, you'll
probably find yourself wanting some functions that don't exist. You might have a specialized need that would benefit
from a specific function, so in Chapter 11 I'll show you how to create functions of your own.

The following sections describe the scalar functions that are supplied with SQL Server 2000. We'll first look at the
CAST function because it's especially important.

Conversion Functions
SQL Server provides three functions for converting datatypes: the generalized CAST function, the CONVERT
function—which is analogous to CAST but has a slightly different syntax—and the more specialized STR function.
NOTE
The CAST function is a synonym for CONVERT and was added to the product to comply with ANSI-92
specifications. You'll see CONVERT used in older documentation and older code.

CAST is possibly the most useful function in all of SQL Server. It allows you to change the datatype when you select
a field, which can be essential for concatenating strings, joining columns that weren't initially envisioned as related,
performing mathematical operations on columns that were defined as character but which actually contain numbers,
and other similar operations. Like C, SQL is a fairly strongly typed language.

Some languages, such as Visual Basic or PL/1 (Programming Language 1), allow you to almost indiscriminately mix
datatypes. If you mix datatypes in SQL, however, you'll often get an error. Some conversions are implicit, so, for
example, you can add or join a smallint and an int without any such error; trying to do the same between an int and
a char, however, produces an error. Without a way to convert datatypes in SQL, you would have to return the
values back to the client application, convert them there, and then send them back to the server. You might need to
create temporary tables just to hold the intermediate converted values. All of this would be cumbersome and
inefficient.

Recognizing this inefficiency, the ANSI committee added the CAST operator to SQL-92. SQL Server's
CAST—and the older CONVERT—provides a superset of ANSI CAST functionality. The CAST syntax is simple:

CAST (original_expression AS desired_datatype)

Suppose I want to concatenate the job_id (smallint) column of the employee table with the lname (varchar)
column. Without CAST, this would be impossible (unless it was done in the application). With CAST, it's trivial:
SELECT lname + '-' + CAST(job_id AS varchar(2)) FROM employee

Here's the abbreviated output:

Accorti-13
Afonso-14
Ashworth-6
Bennett-12
Brown-7
Chang-4
Cramer-2
Cruz-10
Devon-3
This document is created with the unregistered version of CHM2PDF Pilot

Specifying a length shorter than the column or expression in your call to CAST is a useful way to truncate the column.
You could use the SUBSTRING function for this with equal results, but you might find it more intuitive to use CAST.
Specifying a length when you convert to char, varchar, decimal, or numeric isn't required, but it's recommended.
Specifying a length shields you better from possible behavioral changes in newer releases of the product.
NOTE
Although behavioral changes generally don't occur by design, it's difficult to ensure 100 percent consistency of subtle
behavioral changes between releases. The development team strives for such consistency, but it's nearly impossible to
ensure that no behavioral side effects result when new features and capabilities are added. There's little point in relying
on the current default behavior when you can just as easily be explicit about what you expect.

You might want to avoid CAST when you convert float or numeric/decimal values to character strings if you expect
to see a certain number of decimal places. CAST doesn't currently provide formatting capabilities for numbers. The
older CONVERT function has some limited formatting capabilities, which we'll see shortly. Formatting floating-point
numbers and such when converting to character strings is another area that's ripe for subtle behavior differences. So if
you need to transform a floating-point number to a character string in which you expect a specific format, you can use
the other conversion function, STR. STR is a specialized conversion function that always converts from a number (
float, numeric, and so on) to a character datatype, but it allows you to explicitly specify the length and number of
decimal places that should be formatted for the character string. Here's the syntax:

STR(float_expression, character_length, number_of_decimal_places)

Remember that character_length must include room for a decimal point and a negative sign if they might exist. Also
be aware that STR rounds the value to the number of decimal places requested, while CAST simply truncates the
value if the character length is smaller than the size required for full display. Here's an example of STR:
SELECT discounttype, 'Discount'=STR(discount, 7, 3) FROM discounts

And here's the output:

discounttype Discount
------------ --------
Initial Customer 10.500
Volume Discount 6.700
Customer Discount 5.000

You can think of the ASCII and CHAR functions as special type-conversion functions, but I'll discuss them later in
this chapter along with string functions.

If all you're doing is converting an expression to a specific datatype, you can actually use either CAST or the older
CONVERT. However, the syntax of CONVERT is slightly different and allows for a third, optional argument. Here's
the syntax for CONVERT:

CONVERT (desired_datatype[(length)], original_expression

[, style])

CONVERT has an optional third parameter, style. This parameter is most commonly used when you convert an
expression of type datetime or smalldatetime to type char or varchar. You can also use it when you convert float,
real, money, or smallmoney to a character datatype.

When you convert a datetime expression to a character datatype, if the style argument isn't specified, the conversion
will format the date with the default SQL Server date format (for example, Oct 3 1999 2:25PM). By specifying a
style type, you can format the output as you want. You can also specify a shorter length for the character buffer being
converted to, in order to perhaps eliminate the time portion or for other reasons. Table 10-8 shows the various values
This document is created with the unregistered version of CHM2PDF Pilot

you can use as the style argument.

Table 10-8. Values for the style argument of the CONVERT function when you convert a datetime expression
to a character expression.
Style Number without Style Number with
Output Type Style
Century (yy) Century (yyyy)
mon dd yyyy hh:miAM (or
_ 0 or 100 Default
PM)
1 101 USA mm/dd/yyyy
2 102 ANSI yyyy.mm.dd
3 103 British/ French dd/mm/yyyy
4 104 German dd.mm.yyyy
5 105 Italian dd-mm-yyyy
6 106 _ dd mon yyyy
7 107 _ mon dd, yyyy
_ 8 or 108 _ hh:mm:ss
Default + mon dd yyyy
_ 9 or 109
milliseconds (or PM) hh:mi:ss:mmmAM
10 110 USA mm-dd-yy
11 111 JAPAN yy/mm/dd
12 112 ISO yymmdd
Europe default + dd mon yyyy
_ 13 or 113
milliseconds hh:mi:ss:mmm (24h)
14 114 _ hh:mi:ss:mmm (24h)
yyyy-mm-dd
20 or 120 ODBC canonical
hh:mi:ss(24h)
ODBC canonical + yyyy-mm-dd
21 or 121
milliseconds hh:mi:ss.mmm(24h)
NOTE
Although SQL Server uses style 0 as the default when converting a datetime value to a character string, SQL Query
Analyzer (and OSQL) use style 121 when displaying a datetime value.

Since we just passed a change in the century, using two-character formats for the year could be a bug in your
application just waiting to happen! Unless you have a compelling reason not to, you should always use the full year (
yyyy) for both the input and output formatting of dates to prevent any ambiguity.

SQL Server has no problem dealing with the year 2000—in fact, the change in century isn't even a boundary
condition in SQL Server's internal representation of dates. But if you represent a year by specifying only the last two
digits, the inherent ambiguity might cause your application to make an incorrect assumption. When a date formatted
that way is entered, SQL Server's default behavior is to interpret the year as 19yy if the value is greater than or equal
to 50 and as 20yy if the value is less than 50. That might be OK now, but by the year 2050 you won't want a
two-digit year of 50 to be interpreted as 1950 instead of 2050. (If you simply assume that your application will have
been long since retired, think again. A lot of COBOL programmers in the 1960s didn't worry about the year 2000.)

SQL Server 2000 allows you to change the cutoff year that determines how a two-digit year is interpreted. A
two-digit year that is less than or equal to the last two digits of the cutoff year is in the same century as the cutoff year.
A twodigit year that is greater than the last two digits of the cutoff year is in the century that precedes the cutoff year.
You can change the cutoff year by using the Properties dialog box for your SQL server in SQL Server Enterprise
Manager and selecting the Server Settings tab. Or you can use the sp_configure stored procedure:
This document is created with the unregistered version of CHM2PDF Pilot

EXEC sp_configure 'two digit year cutoff', '2000'

In this example, since two digit year cutoff is 2000, all two-digit years other than 00 are interpreted as occurring in
the 20th century. With the default two digit year cutoff value of 2049, the two-digit year 49 is interpreted as 2049
and the two-digit year 50 is interpreted as 1950. You should be aware that although SQL Server uses 2049 as the
cutoff year for interpreting dates, OLE Automation objects use 2030. You can use the two digit year cutoff option
to provide consistency in date values between SQL Server and client applications. However, to avoid ambiguity with
dates, you should always use four-digit years in your data.

You don't need to worry about how changing the two digit year cutoff value will affect your existing data. If data is
stored using the datetime or smalldatetime datatypes, the full four-digit year is part of the internal storage. The two
digit year cutoff value controls only how SQL Server interprets date constants such as 10/12/99.

Day First or Month First

The two digit year cutoff value determines how SQL Server interprets the 99 in 10/12/99, but this date constant
has another ambiguity. Are we talking about October 12 or December 10? The SET option DATEFORMAT
determines whether SQL Server interprets a numeric three-part date as month followed by day, day followed by
month, or even year followed by month. The default value of DATEFORMAT is controlled by the language you're
using, but you can change it using the SET DATEFORMAT command. There are six possible values for
DATEFORMAT, which should be self-explanatory: mdy, dmy, ymd, ydm, dym, and myd. Only the first three are
commonly used. Note that DATEFORMAT controls only how SQL Server interprets date constants that are entered
by you or your program. It does not control how date values are displayed. As I've mentioned, output format is
controlled by the client or by having SQL Server convert the datetime value to a character string and using a style
option.

Here's an example that shows how changing the DATEFORMAT value can change how input dates are interpreted:
SET DATEFORMAT mdy
SELECT 'FORMAT is mdy' = CONVERT(datetime, '7/4/2000')
SET DATEFORMAT dmy
SELECT 'FORMAT is dmy' = CONVERT(datetime, '7/4/2000')

RESULTS:

FORMAT is mdy
---------------------------
2000-07-04 00:00:00.000

FORMAT is dmy
---------------------------
2000-04-07 00:00:00.000

You might consider using ISO style when you insert dates. This format has the year followed by month and then the
day, with no punctuation. This style is always recognizable, no matter what the SQL Server default language and no
matter what the DATEFORMAT setting. Here's the above conversion command with dates supplied in ISO format:
SET DATEFORMAT mdy
SELECT 'FORMAT is mdy' = CONVERT(datetime, '20000704')
SET DATEFORMAT dmy
SELECT 'FORMAT is mdy' = CONVERT(datetime, '20000704')
RESULTS:
FORMAT is mdy
---------------------------
2000-07-04 00:00:00.000

FORMAT is mdy
This document is created with the unregistered version of CHM2PDF Pilot

---------------------------
2000-07-04 00:00:00.000

Be sure to always put your dates in quotes. If we had left off the quotes, SQL Server would assume we meant the
number 20,000,704 and would interpret that as the number of days after January 1, 1900. It would try to figure out
the date that was that many days after the base date, which is out of the range of possible datetime values that SQL
Server can store.

One other date format can override your default language or DATEFORMAT setting, but it behaves a little
inconsistently. If you enter a date in all numeric format with the year first but you include punctuation (as in
1999.05.02), SQL Server will assume that the first part is the year even if your DATEFORMAT is dmy. However,
which number is the month and which is the day? SQL Server will still use your DATEFORMAT setting to determine
the order of the month and the day values. So if your DATEFORMAT is mdy, SELECT CONVERT(datetime,
'1999.5.2') will return 1999-05-02 00:00:00.000, and if your DATEFORMAT is dmy, SELECT
CONVERT(datetime, '1999.5.2') will return 1999-02-05 00:00:00.000 and SELECT CONVERT(datetime,
'1999.5.22') will return an out-of-range error. To me, this seems quite inconsistent. SQL Server is partially ignoring
the DATEFORMAT, but not completely. I suggest that you avoid this format for inputting dates, and if you're going
to use all numbers, leave out the punctuation.

Instead of setting the DATEFORMAT value, you can use the style argument when you convert from a character
string to a datetime value. You must be sure to enter the date in the format that the style requires, as indicated in
Table 10-8 (shown earlier). These two statements return different values:
SELECT CONVERT(datetime, '10.12.99',1)
SELECT CONVERT(datetime, '10.12.99',4)

The first statement assumes that the date is style 1, mm.dd.yy, so the string can be converted to the corresponding
datetime value and returned to the client. The client then returns the datetime value according to its own display
conventions. The second statement assumes that the date is represented as dd.mm.yy. If I had tried to use style 102
or 104, both of which require a four-digit year, I would have received a conversion error because I specified only a
two-digit year.

CONVERT can also be useful if you insert the current date using the GETDATE function but don't consider the time
elements of the datetime datatype to be meaningful. (Remember that SQL Server doesn't currently have separate
date and time datatypes.) You can use CONVERT to format and insert datetime data with only a date element.
Without the time specification in the GETDATE value, the time will consistently be stored as 12:00AM for any given
date. That eliminates problems that can occur when you search among columns or join columns of datetime. For
example, if the date elements are equal between columns but the time element is not, an equality search will fail.

You can eliminate this equality problem by making the time portion consistent. Consider this:
CREATE TABLE my_date (Col1 datetime)
INSERT INTO my_date VALUES (CONVERT(char(10), GETDATE(), 112))

In this example, the CONVERT function converts the current date and time value to a character string that doesn't
include the time. However, because the table we're inserting into expects a datetime value, the new character string is
converted back to a datetime datatype using the default time of midnight.

You can also use the optional style argument when you convert float, real, money, or smallmoney to a character
datatype. When you convert from float or real, the style allows you to force the output to appear in scientific notation
and also allows you to specify either 8 or 16 digits. When you convert from a money type to a character type, the
style allows you to specify whether you want a comma to appear every three digits and whether you want two digits
to the right of the decimal point or four.
This document is created with the unregistered version of CHM2PDF Pilot

In Table 10-9, the column on the left represents the style values for float or real conversion to character data.

Table 10-9. Values for the style argument of the CONVERT function when you convert a floating-point
expression to a character datatype.
Style Value Output
Six digits maximum. In scientific notation when
0 (the default)
appropriate.
1 Always eight digits. Always scientific notation.
2 Always sixteen digits. Always scientific notation.

In Table 10-10, the column on the left represents the style value for money or smallmoney conversion to character
data.

Table 10-10. Values for the style argument of the CONVERT function when you convert a money expression
to a character datatype.
Style Value Output
No commas to the left of the decimal point. Two digits
0 (the default) appear to the right of the decimal point. Example:
4235.98.
Commas every three digits to the left of the decimal point.
1 Two digits appear to the right of the decimal point.
Example: 3,510.92.
No commas to the left of the decimal point. Four digits
2 appear to the right of the decimal point. Example:
4235.9819.

Here's an example from the pubs database. The advance column in the titles table lists the advance paid by the
publisher for each book, and most of the amounts are greater than $1000. In the titles table, the value is stored as a
money datatype. The following query returns the value of advance as a money value, a varchar value with the
default style, and as varchar with each of the two optional styles:
SELECT "Money" = advance,
"Varchar" = CONVERT(varchar(10), advance),
"Varchar-1" = CONVERT(varchar(10), advance, 1),
"Varchar-2" = CONVERT(varchar(10), advance, 2)
FROM titles

Here's the abbreviated output:

Money Varchar Varchar-1 Varchar-2
--------------------- ---------- ---------- ----------
5000.0000 5000.00 5,000.00 5000.0000
5000.0000 5000.00 5,000.00 5000.0000
10125.0000 10125.00 10,125.00 10125.0000
5000.0000 5000.00 5,000.00 5000.0000
.0000 0.00 0.00 0.0000
15000.0000 15000.00 15,000.00 15000.0000
NULL NULL NULL NULL
7000.0000 7000.00 7,000.00 7000.0000
8000.0000 8000.00 8,000.00 8000.0000

Be aware that when you use CONVERT, the conversion occurs on the server and the value is sent to the client
application in the converted datatype. Of course, it's also common for applications to convert between datatypes, but
This document is created with the unregistered version of CHM2PDF Pilot

conversion at the server is completely separate from conversion at the client. It's the client that determines how to
display the value returned. In the previous example, the results in the first column have no comma and four decimal
digits. This is how SQL Query Analyzer displays values of type money that it receives from SQL Server. If we run
this same query using the command-line ISQL tool, the (abbreviated) results look like this:
Money Varchar Varchar-1 Varchar-2
-------------------------- ---------- ---------- ----------
5,000.00 5000.00 5,000.00 5000.0000
5,000.00 5000.00 5,000.00 5000.0000
10,125.00 10125.00 10,125.00 10125.0000
5,000.00 5000.00 5,000.00 5000.0000
0.00 0.00 0.00 0.0000
15,000.00 15000.00 15,000.00 15000.0000
(null) (null) (null) (null)
7,000.00 7000.00 7,000.00 7000.0000
8,000.00 8000.00 8,000.00 8000.0000

NOTE
In addition to determining how to display a value of a particular datatype, the client program determines whether the
output should be left-justified or right-justified and how NULLs are displayed. I'm frequently asked how to change
this default output to, for example, print money values with four decimal digits in SQL Query Analyzer. Unfortunately,
SQL Query Analyzer has its own set of predefined formatting specifications, and, for the most part, you can't change
them. Other client programs, such as report writers, might give you a full range of capabilities for specifying the output
format, but SQL Query Analyzer isn't a report writer.

Here's another example of how the display can be affected by the client. If we use the GETDATE function to select
the current date, it is returned to the client application as a datetime datatype. The client application will likely convert
that internal date representation to a string for display. On the other hand, if we use GETDATE but also explicitly use
CONVERT to change the datetime to character data, the column is sent to the calling application already as a
character string.

So let's see what one particular client will do with that returned data. The command-line program ISQL is a
DB-Library program and uses functions in DBLibrary for binding columns to character strings. DB-Library allows
such bindings to automatically pick up locale styles based on the settings of the workstation running the application.
(You must select the Use International Settings option in the SQL Server Client Network Utility in order for this to
occur by default.) So a column returned internally as a datetime datatype is converted by isql.exe into the same
format as the locale setting of Windows. A column containing a date representation as a character string is not
reformatted, of course.

When we issue the following SELECT statement with SQL Server configured for U.S. English as the default
language and with Windows NT and Windows 2000 on the client configured as English (United States), the date and
string look alike:
SELECT
'returned as date'=GETDATE(),
'returned as string'=CONVERT(varchar(20), GETDATE())

returned as date returned as string

------------------ -------------------
Dec 3 2000 3:10PM Dec 3 2000 3:10PM

But if we change the locale in the Regional Options application of the Windows 2000 Control Panel to French
(Standard), the same SELECT statement returns the following:
returned as date returned as string
----------------- -------------------
3 déc. 2000 15:10 Dec 3 2000 3:10PM
This document is created with the unregistered version of CHM2PDF Pilot

You can see that the value returned to the client in internal date format was converted at the client workstation in the
format chosen by the application. The conversion that uses CONVERT was formatted at the server.
NOTE
Although I've used dates in these examples, this discussion is also relevant to formatting numbers and currency with
different regional settings.

I cannot use the command-line OSQL program to illustrate the same behavior. OSQL, like SQL Query Analyzer, is
an ODBC-based program, and datetime values are converted to ODBC Canonical format, which is independent of
any regional settings. Using OSQL, the previous query would return this result:
returned as date returned as string
--------------------------- --------------------
2000-12-03 15:10:24.793 Dec 3 2000 3:10PM

The rest of the book primarily uses CAST instead of CONVERT when converting between datatypes. CONVERT
is used only when the style argument is needed.

Some conversions are automatic and implicit, so using CAST is unnecessary (but OK). For example, converting
between numbers with types int, smallint, tinyint, float, numeric, and so on happens automatically and implicitly as
long as an overflow doesn't occur, in which case you'd get an error for arithmetic overflow. Converting numbers with
decimal places to integer datatypes results in the truncation of the values to the right of the decimal point—without
warning. (Converting between decimal or numeric datatypes requires that you explicitly use CAST if a loss of
precision is possible.)

Even though Figure 10-3 indicates that conversion from character strings to datetime values can happen implicitly,
this conversion is possible only when SQL Server can figure out what the datetime value needs to be. If you use
wildcards in your character strings, SQL Server might not be able to do a proper conversion.

Suppose I use the following query to find all orders placed in August 1996 and stored in the orders table in the
Northwind database:
USE Northwind
SELECT * FROM orders WHERE OrderDate BETWEEN '8/1/96' and '8/31/96'

Because all the dates have a time of midnight in the orders table, SQL Server will correctly interpret this query and
return 25 rows. It converts the two string constants to datetime values, and then it can do a proper chronological
comparison. However, if your string has a wildcard in it, SQL Server cannot convert it to a datetime, so instead it
converts the datetime into a string. To do this, it uses its default date format, which, as you've seen, is mon dd yyyy
hh:miAM (or PM). So this is the format you have to match in the character string with wildcards. Obviously,
%1996% will match but '8/1/%' won't. Although SQL Server is usually very flexible in how dates can be entered,
once you compare a datetime to a string with wildcards, you can assume only the default format.

Note that in the default format, there are two spaces before a single-digit day. So if I want to find all rows in the
orders table with an OrderDate of July 8, 1996, at any time, I have to use the following and be sure to put two
spaces between Jul and 8:
WHERE OrderDate LIKE 'Jul 8 1996%'

Other conversions, such as between character and integer data, can be performed explicitly only by using CAST or
CONVERT. Conversions between certain datatypes are nonsensical—for example, between a float and a datetime
—and attempting such an operation results in error 529, which states that the conversion is impossible.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 10-3, reprinted from the SQL Server documentation, is one that you'll want to refer to periodically for
conversion issues.

Date and Time Functions

Operations on datetime values are common, such as "get current date and time," "do date arithmetic—50 days from
today is what date," or "find out what day of the week falls on a specific date." Programming languages such as C or
Visual Basic are loaded with functions for operations like these. Transact-SQL also provides a nice assortment to
choose from, as shown in Table 10-11.

The datetime parameter is any expression with a SQL Server datetime datatype or one that can be implicitly
converted, such as an appropriately formatted character string like 1999.10.31. The datepart parameter uses the
encodings shown in Table 10-12. Either the full name or the abbreviation can be passed as an argument.

Table 10-11. Date and time functions in Transact-SQL.

Date Function Return Type Description
DATEADD (datepart, number, Produces a date by adding an interval
datetime
datetime) to a specified date
Returns the number of datepart
DATEDIFF (datepart, datetime1,
int "boundaries" crossed between two
datetime2)
specified dates
Returns a character string repre
DATENAME (datepart, datetime) varchar senting the specified datepart of the
specified date
Returns an integer representing the
DATEPART (datepart, datetime) int specified datepart of the specified
date
Returns an integer representing the
DAY (datetime) int
day datepart of the specified date
Returns an integer representing the
MONTH (datetime) int
month datepart of the specified date
Returns an integer representing the
YEAR (datetime) int
year datepart of the specified date
Returns the current system date and
GETDATE datetime time in the SQL Server standard
internal format for datetime values
Returns the current Universal Time
Coordinate (Greenwich Mean Time),
which is derived from the current local
GETUTCDATE datetime
time and the time zone setting in the
operating system of the machine SQL
Server is running on

Table 10-12. Values for the datepart parameter.

datepart (Full Name) Abbreviation Values
year Yy 1753—9999
quarter Qq 1—4
month Mm 1—12
dayofyear Dy 1—366
day Dd 1—31
This document is created with the unregistered version of CHM2PDF Pilot

week Wk 1—53
weekday Dw 1—7 (Sunday-Saturday)
hour Hh 0—23
minute Mi 0—59
second Ss 0—59
millisecond Ms 0—999

Like other functions, the date functions provide more than simple convenience. Suppose we need to find all records
entered on the second Tuesday of every month and all records entered within 48 weeks of today. Without SQL
Server date functions, the only way we could accomplish such a query would be to select all the rows and return them
to the application and then filter them there. With a lot of data and a slow network, this can get ugly. With the date
functions, the process is simple and efficient, and only the rows that meet these criteria are selected. For this example,
let's assume that the records table includes these columns:
Record_number int
Entered_on datetime

This query returns the desired rows:

SELECT Record_number, Entered_on
FROM records
WHERE
DATEPART(WEEKDAY, Entered_on) = 3
-- Tuesday is 3rd day of week (in USA)
AND
DATEPART(DAY, Entered_on) BETWEEN 8 AND 14
-- The 2nd week is from the 8th to 14th
AND
DATEDIFF(WEEK, Entered_on, GETDATE()) <= 48
-- Within 48 weeks of today

NOTE
The day of the week considered "first" is locale-specific and depends on the DATEFIRST setting.

SQL Server 2000 also allows you to add or subtract an integer to or from a datetime value. This is actually just a
shortcut for the DATEADD function, with a datepart of day. For example, the following two queries are equivalent.
They each return the date 14 days from today:
SELECT DATEADD(day, 14, GETDATE())

SELECT GETDATE() + 14

The date functions don't do any rounding. The DATEDIFF function just subtracts the components from each date
that correspond to the datepart specified. For example, to find the number of years between New Year's Day and
New Year's Eve of the same year, this query would return a value of 0. Because the datepart specifies years, SQL
Server subtracts the year part of the two dates, and because they're the same, the result of subtracting them is 0:
SELECT DATEDIFF(yy, 'Jan 1, 1998', 'Dec 31, 1998')

However, if we want to find the difference in years between New Year's Eve of one year and New Year's Day (the
next day), the following query would return a value of 1 because the difference in the year part is 1:
SELECT DATEDIFF(yy, 'Dec 31, 1998', 'Jan 1, 1999')
This document is created with the unregistered version of CHM2PDF Pilot

There's no built-in mechanism for determining whether two date values are actually the same day unless you've forced
all datetime values to use the default time of midnight. (We saw a technique for doing this earlier.) If you want to
compare two datetime values (@date1 and @date2) to determine whether they're on the same day, regardless of the
time, one technique is to use a three-part condition like this:
IF (DATEPART(mm, @date1) = DATEPART(mm, @date2)) AND
(DATEPART(dd, @date1) = DATEPART(dd, @date2)) AND
(DATEPART(yy, @date1) = DATEPART(yy, @date2))
PRINT 'The dates are the same'

But there is a much simpler way. Even though these two queries both return the same dd value, if we try to find the
difference in days between these two dates, SQL Server is smart enough to know that they are different dates:
SELECT DATEPART(dd, '7/5/99')
SELECT DATEPART(dd, '7/5/00')

So, the following query returns the message that the dates are different:
IF DATEDIFF(dd, '7/5/99','7/5/00') = 0
PRINT 'The dates are the same'
ELSE PRINT 'The dates are different'

And the following general form allows us to indicate whether two dates are really the same date, irrespective of the
time:
IF DATEDIFF(day, @date1, @date2) = 0
PRINT 'The dates are the same'

Math Functions
Transact-SQL math functions are straightforward and typical. Many are specialized and of the type you learned
about in trigonometry class. If you don't have an engineering or mathematical application, you probably won't use
those. A handful of math functions are useful in general types of queries in applications that aren't mathematical in
nature. ABS, CEILING, FLOOR, and ROUND are the most useful functions for general queries to find values within
a certain range.

The random number function, RAND, is useful for generating test data or conditions. You'll see examples of RAND
later in this chapter.

Table 10-13 shows the complete list of math functions and a few simple examples. Some of the examples use other
functions in addition to the math ones to illustrate how to use functions within functions.

Table 10-13. Math functions in Transact-SQL.

Function Parameters Result
Absolute value of the numeric
ABS (numeric_expr) expression. Results returned are of the
same type as numeric_expr.
Angle (in radians) whose cosine is the
ACOS (float_expr) specified approximate numeric (float)
expression.
This document is created with the unregistered version of CHM2PDF Pilot

Angle (in radians) whose sine is the

ASIN (float_expr) specified approximate numeric (float)
expression.
Angle (in radians) whose tangent is the
ATAN (float_expr) specified approximate numeric (float)
expression.
Angle (in radians) whose tangent is (
float_expr1/float_expr2) between
ATN2 (float_expr1, float_expr2)
two approximate numeric (float)
expressions.
Smallest integer greater than or equal
to the numeric expression. Result
CEILING (numeric_expr)
returned is the integer portion of the
same type as numeric_expr.
Trigonometric cosine of the specified
COS (float_expr) angle (in radians) in an approximate
numeric (float) expression.
Trigonometric cotangent of the
specified angle (in radians) in an
COT (float_expr)
approximate numeric (float)
expression.
Degrees converted from radians of the
DEGREES (numeric_expr) numeric expression. Results are of the
same type as numeric_expr.
Exponential value of the specified
EXP (float_expr) approximate numeric (float)
expression.
Largest integer less than or equal to
the specified numeric expression.
FLOOR (numeric_expr)
Result is the integer portion of the
same type as numeric_expr.
Natural logarithm of the specified
LOG (float_expr) approximate numeric (float)
expression.
Base-10 logarithm of the specified
LOG10 (float_expr) approximate numeric (float)
expression.
Constant value of
PI ()
3.141592653589793.
Value of numeric expression to the
power of y, where y is a numeric
datatype (bigint, decimal, float, int,
POWER (numeric_expr, y)
money, numeric, real, smallint,
smallmoney, or tinyint). Result is of
the same type as numeric_expr.
Radians converted from degrees of
RADIANS (numeric_expr) the numeric expression. Result is of
the same type as numeric_expr.
This document is created with the unregistered version of CHM2PDF Pilot

Random approximate numeric (float)

value between 0 and 1, optionally
RAND ([seed])
specifying an integer expression as the
seed.
Numeric expression rounded off to
the length (or precision) specified as
an integer expression (bigint, tinyint,
smallint, or int). Result is of the same
type as numeric_expr. ROUND
always returns a value even if length
ROUND (numeric_expr, length) is illegal. If the specified length is
positive and longer than the digits after
the decimal point, 0 is added after the
fraction digits. If the length is negative
and larger than or equal to the digits
before the decimal point, ROUND
returns 0.00.
Returns the positive (+1), zero (0), or
negative (-1) sign of the numeric
SIGN (numeric_expr)
expression. Result is of the same type
as numeric_expr.
Trigonometric sine of the specified
angle (measured in radians) in an
SIN (float_expr)
approximate numeric (float)
expression.
Square root of the specified
SQRT (float_expr) approximate numeric (float)
expression.
Square of the specified approximate
SQUARE (float_expr)
numeric (float) expression.
Trigonometric tangent of the specified
angle (measured in radians) in an
TAN (float_expr)
approximate numeric (float)
expression.

The following examples show some of the math functions at work.

EXAMPLE 1

Produce a table with the sine, cosine, and tangent of all angles in multiples of 10, from 0 through 180. Format the
return value as a string of eight characters, with the value rounded to five decimal places. (We need one character for
the decimal point and one for a negative sign, if needed.)
DECLARE @degrees smallint
DECLARE @radians float
SELECT @degrees=0
SELECT @radians=0
WHILE (@degrees <= 180)
BEGIN
SELECT
DEGREES=@degrees,
RADIANS=STR(@radians, 8, 5),
SINE=STR(SIN(@radians), 8, 5),
COSINE=STR(COS(@radians), 8, 5),
TANGENT=STR(TAN(@radians), 8, 5)
This document is created with the unregistered version of CHM2PDF Pilot

SELECT @degrees=@degrees + 10
SELECT @radians=RADIANS(CONVERT(float, @degrees))
END

NOTE
This example actually produces 19 different result sets because the SELECT statement is issued once for each
iteration of the loop. These separate result sets are concatenated and appear as one result set in this example. That
works fine for illustrative purposes, but you should avoid doing an operation like this in the real world, especially on a
slow network. Each result set carries with it metadata to describe itself to the client application. I'll show you a better
technique later in the chapter.

And here are the results concatenated as a single table:

DEGREES RADIANS SINE COSINE TANGENT
------- ------- ------- ------- -------
0 0.00000 0.00000 1.00000 0.00000

10 0.17453 0.17365 0.98481 0.17633

20 0.34907 0.34202 0.93969 0.36397

30 0.52360 0.50000 0.86603 0.57735

40 0.69813 0.64279 0.76604 0.83910

50 0.87266 0.76604 0.64279 1.19175

60 1.04720 0.86603 0.50000 1.73205

70 1.22173 0.93969 0.34202 2.74748

80 1.39626 0.98481 0.17365 5.67128

90 1.57080 1.00000 0.00000 *******

100 1.74533 0.98481 -0.1736 -5.6713

110 1.91986 0.93969 -0.3420 -2.7475

120 2.09440 0.86603 -0.5000 -1.7321

130 2.26893 0.76604 -0.6428 -1.1918

140 2.44346 0.64279 -0.7660 -0.8391

150 2.61799 0.50000 -0.8660 -0.5774

160 2.79253 0.34202 -0.9397 -0.3640

170 2.96706 0.17365 -0.9848 -0.1763

3.14159 0.00000 -1.0000 -0.0000

EXAMPLE 2

Express in whole-dollar terms the range of prices (non-null) of all books in the titles table. This example combines
the scalar functions FLOOR and CEILING inside the aggregate functions MIN and MAX:
SELECT 'Low End'=MIN(FLOOR(price)),
'High End'=MAX(CEILING(price))
FROM titles
This document is created with the unregistered version of CHM2PDF Pilot

And the result:

Low End High End
------- --------
2.00 23.00

EXAMPLE 3

Use the same records table that was used in the earlier date functions example. Find all records within 150 days of
September 30, 1997. Without the absolute value function ABS, you would have to use BETWEEN or provide two
search conditions and AND them to account for both 150 days before and 150 days after that date. ABS lets you
easily reduce that to a single search condition.
SELECT Record_number, Entered_on
FROM records
WHERE
ABS(DATEDIFF(DAY, Entered_on, '1997.09.30')) <= 150
-- Plus or minus 150 days

String Functions
String functions make it easier to work with character data. They let you slice and dice character data, search it,
format it, and alter it. Like other scalar functions, string functions allow you to perform functions directly in your search
conditions and SQL batches that would otherwise need to be returned to the calling application for further processing.
(Also, remember the concatenation operator, +, for concatenating strings. You'll use this operator often in conjunction
with the string functions.)

The string functions appear in Table 10-14. For more detailed information, consult the online documentation.

Table 10-14. String functions in Transact-SQL.

Function Parameters Result
Indicates the numeric code value of
ASCII (char_expr) the leftmost character of a character
expression.
Returns the character represented by
the ASCII code. The ASCII code
CHAR (integer_expr)
should be a value from 0 through 255;
otherwise, NULL is returned.
Returns the starting position of the
specified exact pattern. A pattern is
a char_expr. The second parameter
CHARINDEX ('pattern', expression)
is an expression, usually a column
name, in which SQL Server searches
for pattern.
Shows the difference between the
values of two character expressions as
returned by the SOUNDEX function.
DIFFERENCE (char_expr1, char_expr2) DIFFERENCE compares two strings
and evaluates the similarity between
them, returning a value 0 through 4.
The value 4 is the best match.
This document is created with the unregistered version of CHM2PDF Pilot

Returns int_expression characters

LEFT (char_expression, from the int_expression) left of the
char_ expression.
Returns the number of characters,
rather than the number of bytes, of
LEN (char_expression)
char_expression, excluding trailing
blanks.
Converts uppercase character data to
LOWER (char_expr)
lowercase.
LTRIM (char_expr) Removes leading blanks.
Returns the Unicode character with
NCHAR (int_expression) code int_expression, as defined by
the Unicode standard.
Returns the starting position of the first
occurrence of pattern in the specified
PATINDEX ('%pattern%', expression)
ex- pression, or 0 if the pattern isn't
found.
Returns a Unicode string with
quote_character used as the
QUOTENAME ('char_string' [,'
QUOTENAME delimiter to make the input string a
quote_character'])
valid SQL Server delim- ited
identifier.
Replaces all occurrences of
('char_expression1', '
char_expression2 in
REPLACE char_expression2', '
char_expression1 with
char_expression3')
char_expression3.
Repeats a character expression a
specified number of times. If
REPLICATE (char_expr, integer_expr)
integer_expr is negative, NULL is
returned.
Returns the char_expr backwards.
REVERSE (char_expr) This function takes a constant,
variable, or column as its parameter.
Returns part of a character string
starting integer_expr characters from
RIGHT (char_expr, integer_expr)
the right. If integer_expr is negative,
NULL is returned.
RTRIM (char_expr) Removes trailing blanks.
Returns a four-character
(SOUNDEX) of two strings. The
SOUNDEX function converts an
SOUNDEX (char_expr)
alpha string to a four-digit code used
to find similar-sounding words or
names.
Returns a string of repeated spaces.
The number of spaces is equal to
SPACE (integer_expr)
integer_expr. If integer_expr is
negative, NULL is returned.
This document is created with the unregistered version of CHM2PDF Pilot

Returns character data converted

from numeric data. The length is the
total length, including decimal point,
STR (float_expr [, length [,decimal]])
sign, digits, and spaces. The decimal
value is the number of spaces to the
right of the decimal point.
Deletes length characters from
(char_expr1, start, length,
STUFF char_expr1 at start and then inserts
char_expr2)
char_expr2 into char_expr1 at start.
Returns part of a character or binary
string. The first parameter can be a
character or binary string, a column
name, or an expression that includes a
SUBSTRING (expression, start, length)
column name. The second parameter
specifies where the substring begins.
The third parameter specifies the
number of characters in the substring.
Returns the integer value, as defined
UNICODE ('nchar_expression') by the Unicode standard, for the first
character of nchar_expression.
Converts lowercase character data to
UPPER (char_expr)
uppercase.

ASCII and CHAR functions

The function name ASCII is really a bit of a misnomer. ASCII is only a 7-bit character set and hence can deal with
only 128 characters. The character parameter to this function doesn't need to be an ASCII character. The ASCII
function returns the code point for the character set associated with the database or the column if the argument to the
function is a column from a table. The return value can be from 0 through 255. For example, if the current database is
using a Latin-1 character set, the following statement returns 196, even though the character Ä isn't an ASCII
character.
SELECT ASCII('Ä')

The CHAR function is handy for generating test data, especially when combined with RAND and REPLICATE.
CHAR is also commonly used for inserting control characters such as tabs and carriage returns into your character
string. Suppose, for example, that we want to return authors' last names and first names concatenated as a single field,
but with a carriage return (0x0D, or decimal 13) separating them so that without further manipulation in our
application, the names occupy two lines when displayed. The CHAR function makes this simple:
SELECT 'NAME'=au_fname + CHAR(13) + au_lname
FROM authors

Here's the abbreviated result:

NAME
--------
Johnson
White
Marjorie
Green
Cheryl
Carson
Michael
This document is created with the unregistered version of CHM2PDF Pilot

O'Leary
Dean
Straight
Meander
Smith
Abraham
Bennet
Ann
Dull

UPPER and LOWER functions

These functions are useful if you must perform case-insensitive searches on data that is case sensitive. The following
example copies the authors table from the pubs database into a new table, using a case-sensitive collation for the
authors' name fields.
SELECT au_id,
au_lname = au_lname collate Latin1_General_CS_AS,
au_fname = au_fname collate Latin1_General_CS_AS,
phone, address, city, state, zip, contract
INTO authors_CS
FROM authors

This query finds no rows in the new table even though author name Cheryl Carson is included in that table:
SELECT COUNT(*) FROM authors_CS WHERE au_lname='CARSON'

If we change the query to the following, the row will be found:

SELECT COUNT(*) FROM authors_CS WHERE UPPER(au_lname)='CARSON'

If the value to be searched might be of mixed case, you need to use the function on both sides of the equation. This
query will find the row in the case-sensitive table:
DECLARE @name_param varchar(30)
SELECT @name_param='cArSoN'
SELECT COUNT(*) FROM authors_CS WHERE UPPER(au_lname)=UPPER(@name_param)

In these examples, even though we might have an index on au_lname, it won't be useful because the index keys
aren't uppercase. However, we could create a computed column on UPPER(au_lname) and build an index on the
computed column, as described in Chapter 8. In Chapter 15, I'll show you how you can tell if such an index is being
used.

You'll also often want to use UPPER or LOWER in stored procedures in which character parameters are used. For
example, in a procedure that expects Y or N as a parameter, you'll likely want to use one of these functions in case y
is entered instead of Y.

TRIM functions

The functions LTRIM and RTRIM are handy for dealing with leading or trailing blanks. Recall that by default (and if
you don't enable the ANSI_PADDING setting) a varchar datatype is automatically right-trimmed of blanks, but a
fixed-length char isn't. Suppose that we want to concatenate the type column and the title_id column from the titles
table, with a colon separating them but with no blanks. The following query doesn't work because the trailing blanks
are retained from the type column:
This document is created with the unregistered version of CHM2PDF Pilot

SELECT type + ':' + title_id FROM titles

Here's the result (in part):

business :BU1032
business :BU1111
business :BU2075
business :BU7832
mod_cook :MC2222
mod_cook :MC3021
UNDECIDED :MC3026
popular_comp:PC1035
popular_comp:PC8888
popular_comp:PC9999

But RTRIM returns what we want:

SELECT RTRIM(type) + ':' + title_id FROM titles

And here's the result (in part):

business:BU1032
business:BU1111
business:BU2075
business:BU7832
mod_cook:MC2222
mod_cook:MC3021
UNDECIDED:MC3026
popular_comp:PC1035
popular_comp:PC8888
popular_comp:PC9999

String manipulation functions

Functions are useful for searching and manipulating partial strings that include SUBSTRING, CHARINDEX,
PATINDEX, STUFF, REPLACE, REVERSE, and REPLICATE. CHARINDEX and PATINDEX are similar, but
CHARINDEX demands an exact match and PATINDEX works with a regular expression search.
NOTE
You can also use CHARINDEX or PATINDEX as a replacement for LIKE. For example, instead of saying
WHERE name LIKE '%Smith%', you can say WHERE CHARINDEX('Smith', name) > 0.

Suppose I want to change occurrences of the word "computer" within the notes field of the titles table and replace it
with "Hi-Tech Computers." Assume that SQL Server is case sensitive and, for simplicity, that I know that "computer"
won't appear more than once per column. I want to be careful that the plural form, "computers," isn't also changed. I
can't just rely on searching on "computer" with a trailing blank because the word might fall at the end of the sentence
followed by a period or be followed by a comma. The regular expression computer[^s] always finds the word
"computer" and ignores "computers," so it'll work perfectly with PATINDEX.

Here's the data before the change:

SELECT title_id, notes
FROM titles
WHERE notes LIKE '%[Cc]omputer%'

title_id notes
This document is created with the unregistered version of CHM2PDF Pilot

-------- ------------------------------------------------------
BU7832 Annotated analysis of what computers can do for you: a
no-hype guide for the critical user.
PC8888 Muckraking reporting on the world's largest computer
hardware and software manufacturers.
PC9999 A must-read for computer conferencing.
PS1372 A must for the specialist, this book examines the
difference between those who hate and fear computers
and those who don't.
PS7777 Protecting yourself and your loved ones from undue
emotional stress in the modern world. Use of computer
and nutritional aids emphasized.

You might consider using the REPLACE function to make the substitution. However, REPLACE requires that we
search for a specific string that can't include wildcards like [^s]. Instead, we can use the older STUFF function, which
has a slightly more complex syntax. STUFF requires that we specify the starting location within the string to make a
substitution. We can use PATINDEX to find the correct starting location, and PATINDEX allows wildcards:
UPDATE titles
SET notes=STUFF(notes, PATINDEX('%computer[^s]%', notes),
DATALENGTH('computer'), 'Hi-Tech Computers')
WHERE PATINDEX('%computer[^s]%', notes) > 0

Here's the data after the change:

SELECT title_id, notes
FROM titles
WHERE notes LIKE '%[Cc]omputer%'

title_id notes
-------- ------------------------------------------------------
BU7832 Annotated analysis of what computers can do for you: a
no-hype guide for the critical user.
PC8888 Muckraking reporting on the world's largest Hi-Tech
Computers hardware and software manufacturers.
PC9999 A must-read for Hi-Tech Computers conferencing.
PS1372 A must for the specialist, this book examines the
difference between those who hate and fear computers
and those who don't.
PS7777 Protecting yourself and your loved ones from undue
emotional stress in the modern world. Use of Hi-Tech
Computers and nutritional aids emphasized.

Of course, we could have simply provided 8 as the length parameter of the string "computer" to be replaced. But we
used yet another function, LEN, which would be more realistic if we were creating a general-purpose,
search-and-replace procedure. Note that DATALENGTH returns NULL if the expression is NULL, so in your
applications, you might go one step further and use DATALENGTH inside the ISNULL function to return 0 in the
case of NULL.

The REPLICATE function is useful for adding filler characters—such as for test data. (In fact, generating test data
seems to be about the only practical use for this function.) The SPACE function is a special-purpose version of
REPLICATE: it's identical to using REPLICATE with the space character. REVERSE reverses a character string.
You can use REVERSE in a few cases to store and, more importantly, to index a character column backward to get
more selectivity from the index. But in general, these three functions are rarely used.

SOUNDEX and DIFFERENCE Functions

This document is created with the unregistered version of CHM2PDF Pilot

If you've ever wondered how telephone operators are able to give you telephone numbers so quickly when you call
directory assistance, chances are they're using a SOUNDEX algorithm. The SOUNDEX and DIFFERENCE
functions let you search on character strings that sound similar when spoken. SOUNDEX converts each string to a
four-character code. DIFFERENCE can then be used to evaluate the level of similarity between the SOUNDEX
values for two strings as returned by SOUNDEX. For example, you could use these functions to look at all rows that
sound like "Erickson," and they would find "Erickson," "Erikson," "Ericson," "Ericksen," "Ericsen," and so on.

SOUNDEX algorithms are commonplace in the database business, but the exact implementation can vary from
vendor to vendor. For SQL Server's SOUNDEX function, the first character of the four-character SOUNDEX code
is the first letter of the word, and the remaining three characters are single digits that describe the phonetic value of the
first three consonants of the word with the underlying assumption that there's one consonant per syllable. Identical
consonants right next to each other are treated as a single consonant. Only nine phonetic values are possible in this
scheme because only nine possible digits exist. SOUNDEX relies on phonetic similarities between sounds to group
consonants together. In fact, SQL Server's SOUNDEX algorithm uses only seven of the possible nine digits. No
SOUNDEX code uses the digits 8 or 9.

Vowels are ignored, as are the characters h and y, so "Zastrak" would have the same code as "Zasituryk." If no
second or subsequent consonant exists, the phonetic value for it is 0. In addition, SQL Server's SOUNDEX algorithm
stops evaluating the string as soon as the first nonalphabetic character is found. So if you have a hyphenated or
two-word name (with a space in between), SOUNDEX ignores the second part of the name. More seriously,
SOUNDEX stops processing names as soon as it finds an apostrophe. So "O'Flaherty," "O'Leary," and "O'Hara" all
have the same SOUNDEX code, namely O000.

DIFFERENCE internally compares two SOUNDEX values and returns a score from 0 through 4 to indicate how
close the match is. A score of 4 is the best match, and 0 means no matches were found. Executing
DIFFERENCE(a1, a2) first generates the four-character SOUNDEX values for a1 (call it sx_a1) and a2 (call it
sx_a2). Then, if all four values of the SOUNDEX value sx_a2 match the values of sx_a1, the match is perfect and
the level is 4. Note that this compares the SOUNDEX values by character position, not by actual characters.

For example, the names "Smythe" and "Smith" both have a SOUNDEX value of S530, so their difference level is 4,
even though the spellings differ. If the first character (a letter, not a number) of the SOUNDEX value sx_a1 is the
same as the first character of sx_a2, the starting level is 1. If the first character is different, the starting level is 0. Then
each character in sx_a2 is successively compared to all characters in sx_a1. When a match is found, the level is
incremented and the next scan on sx_a1 starts from the location of the match. If no match is found, the next sx_a2
character is compared to the entire four-character list of sx_a1.

The preceding description of the algorithms should make it clear that SOUNDEX at best provides an approximation.
Even so, sometimes it works extremely well. Suppose we want to query the authors table for names that sound
similar to "Larsen." We'll define similar to mean "having a SOUNDEX value of 3 or 4":
SELECT au_lname,
Soundex=SOUNDEX(au_lname),
Diff_Larsen=DIFFERENCE(au_lname, 'Larson')
FROM authors
WHERE
DIFFERENCE(au_lname, 'Larson') >= 3

Here's the result:

au_lname Soundex Diff_Larsen
-------- ------- -----------
Carson C625 3
Karsen K625 3

In this case, we found two names that rhyme with "Larsen" and didn't get any bad hits of names that don't seem
This document is created with the unregistered version of CHM2PDF Pilot

close. Sometimes you'll get odd results, but if you follow the algorithms I've described, you'll understand how these
oddities occur. For example, do you think "Bennet" and "Smith" sound similar? Well, SOUNDEX does. When you
investigate, you can see why the two names have a similar SOUNDEX value. The SOUNDEX values have a different
first letter, but the m and n sounds are converted to the same digit, and both names include a t, which is converted to
a digit for the third position. "Bennet" has nothing to put in the fourth position, so it gets a 0. For Smith, the h becomes
a 0. So, except for the initial letter, the rest of the SOUNDEX strings are the same. Hence, the match.

This type of situation happens often with SOUNDEX—you get hits on values that don't seem close, although usually
you won't miss the ones that are similar. So if you query the authors table for similar values (with a DIFFERENCE
value of 3 or 4) as "Smythe," you get a perfect hit on "Smith" as you'd expect, but you also get a close match for
"Bennet," which is far from obvious:
SELECT au_lname,
Soundex=SOUNDEX(au_lname),
Diff_Smythe=DIFFERENCE(au_lname, 'Smythe')
FROM authors
WHERE
DIFFERENC3E(au_lname, 'Smythe') >= 3

au_lname Soundex Diff_Smythe

-------- ------- -----------
Smith S530 4
Bennet B530 3

Sometimes SOUNDEX misses close matches altogether. This happens when consonant blends are present. For
example, you might expect to get a close match between "Knight" and "Nite." But you get a low value of 1 in the
DIFFERENCE between the two:
SELECT
"SX_KNIGHT"=SOUNDEX('Knight'),
"SX_NITE"=SOUNDEX('Nite'),
"DIFFERENCE"=DIFFERENCE('Nite', 'Knight')

SX_KNIGHT SX_NITE DIFFERENCE

--------- ------- ----------
K523 N300 1

System Functions
System functions are most useful for returning certain metadata, security, or configuration settings. Table 10-15 lists
of some of the more commonly used system functions, and it's followed by a brief discussion of the interesting features
of a few of them. For complete details, see the SQL Server online documentation.

Table 10-15. System functions in Transact-SQL.

Function Parameters Description
Returns the program name for the
APP_NAME NONE current connection if one has been set
by the program before you log on.
A specialized form of the CASE
(expression1, expression2, ...
COALESCE statement. Returns the first non-null
expressionN)
expression.
The defined (maximum) storage length
COL_LENGTH ('table_name', 'column_name')
of a column.
COL_NAME (table_id, column_id) The name of the column.
This document is created with the unregistered version of CHM2PDF Pilot

The length of an expression of any

DATALENGTH ('expression')
datatype.
DB_ID (['database_name']) The database identification number.
DB_NAME ([database_id]) The database name.
The default nullability for the database.
GETANSINULL (['database_name']) Returns 1 when the nullability is the
ANSI NULL default.
The process ID of the application
calling SQL Server on the
workstation. (If you look at the PID
HOST_ID NONE column on the Processes tab in
Windows Task Manager, you will see
this number associated with the client
application.)
HOST_NAME NONE The workstation name.
The increment value specified during
creation of an identity column of a
IDENT_INCR ('table_or_view')
table or view that includes an identity
column.
The seed value specified during
creation of an identity column of a
IDENT_SEED ('table_or_view')
table or view that includes an identity
column.
INDEX_COL ('table_name', index_id, key_id) The indexed column name(s).
Checks whether an expression is a
datetime datatype or a string in a
recognizable datetime format.
ISDATE (expression_of_possible_date)
Returns 1 when the expression is
compatible with the datetime type;
otherwise, returns 0.
Replaces NULL entries with the
ISNULL (expression, value)
specified value.
Checks whether an expression is a
numeric datatype or a string in a
recognizable number format. Returns
ISNUMERIC (expression_of_possible_number)
1 when the expression is compatible
with arithmetic operations; otherwise,
returns 0.
A specialized form of CASE. The
resulting expression is NULL when
NULLIF (expression1, expression2)
expression1 is equivalent to
expression2.
The database object identification
OBJECT_ID ('object_name')
number.
OBJECT_NAME (object_id) The database object name.
The date when the statistics for the
STATS_DATE (table_id, index_id) specified index (index_id) were last
updated.
The security identification number
SUSER_SID (['login_name'])
(SID) for the user's login name.
This document is created with the unregistered version of CHM2PDF Pilot

The login identification name from a

SUSER_SNAME ([server_user_sid]) user's security identification number
(SID).
USER_ID (['user_name']) The user's database ID number.
USER_NAME ([user_id]) The user's database username.

The DATALENGTH function is most often used with variable-length data, especially character strings, and it tells you
the actual storage length of the expression (typically a column name). A fixed-length datatype always returns the
storage size of its defined type (which is also its actual storage size), which makes it identical to COL_LENGTH for
such a column. The DATALENGTH of any NULL expression returns NULL. The OBJECT_ID, OBJECT_NAME,
SUSER_SNAME, SUSER_SID, USER_ID, USER_NAME, COL_NAME, DB_ID, and DB_NAME functions
are commonly used to more easily eliminate the need for joins between system catalogs and to get run-time
information dynamically.

For example, recognizing that an object name is unique within a database for a specific user, we can use the following
statement to determine whether an object by the name "foo" exists for our current user ID; if so, we can drop the
object (assuming that we know it's a table).
IF (SELECT id FROM sysobjects WHERE id=OBJECT_ID('foo')
AND uid=USER_ID() AND type='U') > 0
DROP TABLE foo

System functions can be handy with constraints and views. Recall that a view can benefit from the use of system
functions. For example, if the accounts table includes a column that is the system login ID of the user to whom the
account belongs, we can create a view of the table that allows the user to work only with his or her own accounts.
We do this simply by making the WHERE clause of the view something like this:
WHERE system_login_id=SUSER_SID()

Or if we want to ensure that updates on a table occur only from an application named CS_UPDATE.EXE, we can
use a CHECK constraint in the following way:
CONSTRAINT APP_CHECK(APP_NAME()='CS_UPDATE.EXE')

For this particular example, the check is by no means foolproof because the application must set the app_name in its
login record. A rogue application could simply lie. To prevent casual use by someone running an application like
isql.exe, the preceding constraint might be just fine for your needs. But if you're worried about hackers, this
formulation isn't appropriate. Other functions, such as SUSER_SID, have no analogous way to be explicitly set, so
they're better for security operations like this.

The ISDATE and ISNUMERIC functions can be useful for determining whether data is appropriate for an operation.
For example, suppose your predecessor didn't know much about using a relational database and defined a column as
type char and then stored values in it that were naturally of type money. Then she compounded the problem by
sometimes encoding letter codes in the same field as notes. You're trying to work with the data as is (eventually you'll
clean it up, but you have a deadline to meet), and you need to get a sum and average of all the values whenever a
value that makes sense to use is present. (If you think this example is contrived, talk to a programmer in an MIS shop
of a Fortune 1000 company with legacy applications.) Suppose the table has a varchar(20) column named acct_bal
with these values:
acct_bal
--------
205.45
E
(NULL)
This document is created with the unregistered version of CHM2PDF Pilot

B
605.78
32.45
8
98.45
64.23
8456.3

If you try to simply use the aggregate functions directly, you'll get error 409:
SELECT SUM(acct_bal). AVG(acct_bal)
FROM bad_column_for_money

MSG 235, Level 16, State 0

Cannot covert CHAR value to MONEY. The CHAR value has incorrect
syntax.

But if you use both CONVERT() in the select list and ISNUMERIC() in the WHERE clause to choose only those
values for which a conversion would be possible, everything works great:
SELECT
"SUM"=SUM(CONVERT(money, acct_bal)),
"AVG"=AVG(CONVERT(money, acct_bal))
FROM bad_column_for_money
WHERE ISNUMERIC(acct_bal)=1

SUM AVG
-------- --------
9,470 1.352.95

Metadata Functions
In earlier versions of SQL Server, the only way to determine which options or properties an object had was to
actually query a system table and possibly even decode a bitmap status column. SQL Server 2000 provides a set of
functions that return information about databases, files, filegroups, indexes, objects, and datatypes.

For example, to determine what recovery model a database is using, you can use the DATABASEPROPERTYEX
function:
IF DATABASEPROPERTYEX('pubs', 'Recovery') <> 'SIMPLE'
/*run a command to backup the transaction log */

Many of the properties that you can check for using either the DATABASEPROPERTY and
DATABASEPROPERTYEX functions correspond to database options that you can set using the sp_dboption
stored procedure. One example is the AUTO_CLOSE option, which determines the value of the IsAutoClose
property. Other properties, such as IsInStandBy, are set internally by SQL Server. For a complete list of all the
possible database properties and their possible values, please see the SQL Server documentation. The documentation
also provides the complete list of properties and possible values for use with the following functions:
 COLUMNPROPERTY
 FILEGROUPPROPERTY
 FILEPROPERTY
 INDEXPROPERTY
 OBJECTPROPERTY
 TYPEPROPERTY
This document is created with the unregistered version of CHM2PDF Pilot

Niladic Functions
ANSI SQL-92 has a handful of what it calls niladic functions—a fancy name for functions that don't accept any
parameters. Niladic functions were first implemented in version 6, and each one maps directly to one of SQL Server's
system functions. They were actually added to SQL Server 6 for conformance with the ANSI SQL-92 standard; all
of their functionality was already provided, however. Table 10-16 lists the niladic functions and their equivalent SQL
Server system functions.

Table 10-16. Niladic functions and their equivalent SQL Server functions.
Niladic Function Equivalent SQL Server System Function
CURRENT_TIMESTAMP GETDATE
SYSTEM_USER SUSER_SNAME
CURRENT_USER USER_NAME
SESSION_USER USER_NAME
USER USER_NAME

If you execute the following two SELECT statements, you'll see that they return identical results:
SELECT CURRENT_TIMESTAMP, USER, SYSTEM_USER, CURRENT_USER,
SESSION_USER

SELECT GETDATE(), USER_NAME(), SUSER_SNAME(),

USER_NAME(), USER_NAME()

Other Parameterless Functions

As I mentioned earlier in the chapter, the older SQL Server documentation refers to a set of parameterless system
functions as global variables because they're designated by @@, which is similar to the designation for local
variables. However, these values aren't variables because you can't declare them and you can't assign them values.

These functions are global only in the sense that any connection can access their values. However, in many cases the
value returned by these functions is specific to the connection. For example, @@ERROR represents the last error
number generated for a specific connection, not the last error number in the entire system. @@ROWCOUNT
represents the number of rows selected or affected by the last statement for the current connection.

Many of these parameterless system functions keep track of performance-monitoring information for your SQL
Server. These include @@CPU_BUSY, @@IO_BUSY, @@PACK_SENT, and @@PACK_RECEIVED.

Some of these functions are extremely static, and others are extremely volatile. For example, @@version represents
the build number and code freeze date of the SQL Server executable (Sqlservr.exe) that you're running. It will change
only when you upgrade or apply a service pack. Functions like @@ROWCOUNT are extremely volatile. Take a
look at the following code and its results.
USE pubs
SELECT * FROM publishers
SELECT @@ROWCOUNT
SELECT @@ROWCOUNT

Here are the results:

pub_id pub_name city state country
------ ------------------------- -------------------- ----- ----------
0736 New Moon Books Boston MA USA
This document is created with the unregistered version of CHM2PDF Pilot

0877 Binnet & Hardley Washington DC USA

1389 Algodata Infosystems Berkeley CA USA
1622 Five Lakes Publishing Chicago IL USA
1756 Ramona Publishers Dallas TX USA
9901 GGG&G München NULL Germany
9952 Scootney Books New York NY USA
9999 Lucerne Publishing Paris NULL France

(8 row(s) affected)

-----------
8

(1 row(s) affected)

-----------
1

(1 row(s) affected)

Note that the first time @@ROWCOUNT was selected, it returned the number of rows in the publishers table (8).
The second time @@ROWCOUNT was selected, it returned the number of rows returned by the previous SELECT
@@ROWCOUNT (1). Any query you execute that affects rows will change the value of @@ROWCOUNT. For
the complete list of parameterless functions, refer to the SQL Server documentation.

Table-Valued Functions
SQL Server 2000 provides a set of system functions that return tables; because of this, the functions can appear in
the FROM clause of a query. To invoke these functions, you must use the special signifier :: as a prefix to the function
name and then omit any database or owner name. Several of these table-valued functions return information about
traces you have defined, and I'll talk about those functions in Chapter 17 when I discuss SQL Profiler. Another one of
the functions can also be useful during monitoring and tuning, as shown in this example:
SELECT *
FROM ::fn_virtualfilestats(5,1)

The function fn_virtualfilestats takes two parameters, a database ID and a file ID, and it returns statistical
information about the I/O on the file. Here is some sample output from running this query on the pubs database:
Number Number Bytes Bytes
DbId FileId TimeStamp Reads Writes Read Written IoStallMS
---- ------ --------- ------ ------ ------- ------- ---------
5 1 38140863 44 0 360448 0 3164

Because the function returns a table, you can control the result set by limiting it to only certain columns or to rows that
meet certain conditions. As another example, the function fn_helpcollations lists all the collations supported by SQL
Server 2000. An unqualified SELECT from this function would return 753 rows in two columns. If I just want the
names of all the SQL Collations that are Case Insensitive, have no preference, and are based on the Latin1 character
set, I can issue the following query, which returns only 14 rows:
SELECT name
FROM ::fn_helpcollations()
WHERE name LIKE 'SQL%latin%CI%'
AND name NOT LIKE '%pref%'

RESULT:
This document is created with the unregistered version of CHM2PDF Pilot

name
---------------------------------------
SQL_Latin1_General_CP1_CI_AI
SQL_Latin1_General_CP1_CI_AS
SQL_Latin1_General_CP1250_CI_AS
SQL_Latin1_General_CP1251_CI_AS
SQL_Latin1_General_CP1253_CI_AI
SQL_Latin1_General_CP1253_CI_AS
SQL_Latin1_General_CP1254_CI_AS
SQL_Latin1_General_CP1255_CI_AS
SQL_Latin1_General_CP1256_CI_AS
SQL_Latin1_General_CP1257_CI_AS
SQL_Latin1_General_CP437_CI_AI
SQL_Latin1_General_CP437_CI_AS
SQL_Latin1_General_CP850_CI_AI
SQL_Latin1_General_CP850_CI_AS
This document is created with the unregistered version of CHM2PDF Pilot

34

Transact-SQL Examples and

Brainteasers
In this section, we'll look at some relatively common programming tasks that aren't as simple as you might think. If
you find a solution here for something you need to do, that's great. But the real goal is to give you some insight into the
power and flexibility of Transact-SQL. For some examples, I'll offer multiple solutions, often with different
performance characteristics. Keep in mind that there are often many ways to solve a problem, and you can find more
examples similar to the ones I present here in some of the SQL programming books listed in the bibliography.

Generating Test Data

There is actually nothing particularly tricky to generating test data, but because generating test data is a common need
and some of the later solutions depend on test data, we'll look at a few simple techniques. Of course, it's preferable to
use an actual dataset of representative values for testing rather than generated test data before you deploy a
production application. But sometimes it's more practical to generate test data when you need something "quick and
dirty."

If you need a bunch of rows and you don't care about the distribution of values, you can easily create a table using
default values for every column and the Identity property on a column that will act as the primary key. You can then
set a counter to loop as many times as necessary to achieve the number of rows you want and execute the special
INSERT?DEFAULT VALUES statement. This example creates 1000 rows in table xyz:
-- Method 1. Simple DEFAULT values on table.
CREATE TABLE xyz
(
col1 int PRIMARY KEY IDENTITY(1, 1) NOT NULL,
col2 int NOT NULL DEFAULT 999,
col3 char(10) NOT NULL DEFAULT 'ABCEFGHIJK'
)
GO
SET NOCOUNT ON
DECLARE @counter int
SET @counter=1
WHILE (@counter <= 1000)
BEGIN
INSERT xyz DEFAULT VALUES
SET @counter=@counter+1
END

SELECT * FROM xyz

col1 col2 col3

---- ---- -----------
1 999 ABCEFGHIJK
2 999 ABCEFGHIJK
3 999 ABCEFGHIJK
4 999 ABCEFGHIJK
5 999 ABCEFGHIJK

999 999 ABCEFGHIJK

1000 999 ABCEFGHIJK
This document is created with the unregistered version of CHM2PDF Pilot

Usually, you want some distribution in the data values; the RAND function, the modulo operator, and the functions
CONVERT, CHAR, and REPLICATE come in handy for this purpose. RAND is a standard random number
generator that's just like the function used in C, which RAND calls. Because RAND returns a float with a value
between 0 and 1, you typically multiply it and convert the result to an integer so that you can use the modulo operator
to indicate the range of values. For example, if you want a random integer from 0 through 9999, the following
expression works nicely:
(CONVERT(int, RAND() * 100000) % 10000)

If you want to include negative numbers (for example, a range from -9999 through 9999), you can use RAND to flip
a coin and generate 0 or 1 (by doing modulo 2) and then use CASE to multiply half of the numbers by -1:
CASE
WHEN CONVERT(int, RAND() * 1000) % 2 = 1 THEN
(CONVERT(int, RAND() * 100000) % 10000 * -1)
ELSE CONVERT(int, RAND() * 100000) % 10000
END

To produce character data, you can generate a random number from 0 through 25 (since the English alphabet has 26
letters) and add that number to 65, which is the ASCII value for A. The result is the ASCII value for a character
from A through Z. You can perform this operation a specified number of times or in a loop to generate as many
characters as you want. Usually, after you indicate a few lead characters, you can use filler characters for the rest of a
field. The REPLICATE function is a nice tool to use for creating the filler. If you want to be sure that different
executions of the routine return different random numbers, you should seed the RAND function by including an integer
value between the parentheses. You can use @@spid, the object_id of the newly created table, or any other "almost
random" integer value.

The most common way to generate character data is to use local variables for the generated data inside a WHILE
loop that does INSERT statements using the variables. Here's an example:
-- Method 2. Generate random data in a loop.
IF (ISNULL(OBJECT_ID('random_data'), 0)) > 0
DROP TABLE random_data
GO

CREATE TABLE random_data

(
col1 int PRIMARY KEY,
col2 int,
col3 char(15)
)
GO

DECLARE @counter int, @col2 int, @col3 char(15)

/* Insert 100 rows of data */
-- Seed random generator
SELECT @counter=0, @col2=RAND(@@spid + cpu + physical_io)
FROM master..sysprocesses where spid=@@spid

WHILE (@counter < 1000)

BEGIN
SELECT @counter=@counter + 10, -- Sequence numbers by 10
@col2=
CASE -- Random integer between -9999 and 9999
WHEN CONVERT(int, RAND() * 1000) % 2 = 1
THEN (CONVERT(int, RAND() * 100000) % 10000 * -1)
ELSE CONVERT(int, RAND() * 100000) % 10000
This document is created with the unregistered version of CHM2PDF Pilot

END,
@col3= -- Four random letters followed by random fill letter
CHAR((CONVERT(int, RAND() * 1000) % 26 ) + 65) -- 65 is 'A'
+ CHAR((CONVERT(int, RAND() * 1000) % 26 ) + 65)
+ CHAR((CONVERT(int, RAND() * 1000) % 26 ) + 65)
+ CHAR((CONVERT(int, RAND() * 1000) % 26 ) + 65)
+ REPLICATE(CHAR((CONVERT(int, RAND() * 1000) % 26 )
+ 65), 11)

INSERT random_data VALUES (@counter, @col2, @col3)

END
GO

SELECT * FROM random_data WHERE COL1 < 200

col1 col2 col3

---- ----- ---------------
10 -5240 LXDSGGGGGGGGGGG
20 9814 TTPDOOOOOOOOOOO
30 3004 IEYXEEEEEEEEEEE
40 -9377 MITDAAAAAAAAAAA
50 -3823 ISGMUUUUUUUUUUU
60 -4249 DHZQQQQQQQQQQQQ
70 2335 XBJKEEEEEEEEEEE
80 -4570 ILYWNNNNNNNNNNN
90 4223 DHISDDDDDDDDDDD
100 -3332 THXLWWWWWWWWWWW
110 -9959 ALHFLLLLLLLLLLL
120 4580 BCZNGGGGGGGGGGG
130 6072 HRTJOOOOOOOOOOO
140 -8274 QPTKWWWWWWWWWWW
150 8212 FBQABBBBBBBBBBB
160 8223 YXAPLLLLLLLLLLL
170 -9469 LIHCAAAAAAAAAAA
180 -2959 GYKRZZZZZZZZZZZ
190 7677 KWWBJJJJJJJJJJJ

You can also set up the table with a DEFAULT that includes the random data expression and then use the
DEFAULT VALUES statement. This method is a combination of the previous two methods. All the complexity of the
random values is segregated to the CREATE TABLE command and the INSERT is again a simple loop that uses
DEFAULT VALUES. In the following example, we use a CASE statement in a DEFAULT clause of a CREATE
TABLE command:
-- Method 3. Generate random values for DEFAULT.
CREATE TABLE random_data
(
col1 int PRIMARY KEY IDENTITY(10,10) NOT NULL,
col2 int NOT NULL DEFAULT CASE
-- Random integer between -9999 and 9999
WHEN CONVERT(int, RAND() * 1000) % 2 = 1
THEN (CONVERT(int, RAND() * 100000) % 10000 * -1 )
ELSE CONVERT(int, RAND() * 100000) % 10000
END,
col3 char(15) NOT NULL DEFAULT
CHAR((CONVERT(int, RAND() * 1000) % 26 ) + 65)
-- 65 is 'A'
+ CHAR((CONVERT(int, RAND() * 1000) % 26 ) + 65)
+ CHAR((CONVERT(int, RAND() * 1000) % 26 ) + 65)
+ CHAR((CONVERT(int, RAND() * 1000) % 26 ) + 65)
+ REPLICATE(CHAR((CONVERT(int, RAND() * 1000)
% 26) + 65), 11)
)
GO
This document is created with the unregistered version of CHM2PDF Pilot

DECLARE @counter int

SET @counter=1
WHILE (@counter <= 1000)
BEGIN
INSERT random_data DEFAULT VALUES
SET @counter=@counter + 1
END

-- Limit number of rows for illustration only

SELECT * FROM random_data WHERE col1 <= 200

col1 col2 col3

---- ----- ---------------
10 -6358 LCNLMMMMMMMMMMM
20 -2284 SSAITTTTTTTTTTT
30 -1498 NARJAAAAAAAAAAA
40 -1908 EINLZZZZZZZZZZZ
50 -716 KNIOFFFFFFFFFFF
60 -8331 WZPRYYYYYYYYYYY
70 -2571 TMUBEEEEEEEEEEE
80 -7965 LILNCCCCCCCCCCC
90 9728 IXLOBBBBBBBBBBB
100 878 IPMPPPPPPPPPPPP
110 -2649 QXPAPPPPPPPPPPP
120 -4443 EBVHKKKKKKKKKKK
130 6133 VRJWXXXXXXXXXXX
140 -5154 HMHXLLLLLLLLLLL
150 -480 RNLVQQQQQQQQQQQ
160 -2655 SEHXTTTTTTTTTTT
170 -8204 JVLHZZZZZZZZZZZ
180 -3201 PTWGBBBBBBBBBBB
190 -7529 TDCJXXXXXXXXXXX
200 2622 ANLDHHHHHHHHHHH

Getting Rankings
To get the five highest values or five lowest values in a dataset, you can use the TOP clause in a SQL Server
SELECT statement. (I discussed TOP in Chapter 7.) The TOP clause also lets you specify WITH TIES. It's not
always clear what that means, so an example might help. The following SELECT statement returns the rows with the
five highest values for the ytd_sales column, including any ties:
SELECT TOP 5 WITH TIES title_id, ytd_sales, title
FROM titles
ORDER BY ytd_sales DESC

Here's the result:

title_id ytd_sales title
-------- --------- -----
MC3021 22246 The Gourmet Microwave
BU2075 18722 You Can Combat Computer Stress!
TC4203 15096 Fifty Years in Buckingham Palace Kitchens
PC1035 8780 But Is It User Friendly?
BU1032 4095 The Busy Executive's Database Guide
BU7832 4095 Straight Talk About Computers
PC8888 4095 Secrets of Silicon Valley
TC7777 4095 Sushi, Anyone?
(8 row(s) affected)
This document is created with the unregistered version of CHM2PDF Pilot

You end up with more than the five rows you requested. SQL Server returned the first five rows after sorting the
data, and because additional rows had the same value as the fifth row returned (4095), it kept returning rows as long
as there were equivalent ytd_sales values. The WITH TIES clause requires that the data be sorted using ORDER
BY. A more "formal" definition of TOP n WITH TIES and ORDER BY c DESC is "return all rows that have a value
of c for which there are at most n-1 greater values of c." So, in the dataset above, n is 5 and c is ytd_sales. The value
22246 has 0 rows with a greater value of ytd_sales, and 18722 has only one value greater. The last value returned,
4095, has only four values that are greater.

Another way to think of this concept of "the number of distinct values greater than a given value" is as the rank of the
value. A value for which there are no greater values has a rank of 1, and a value for which there are two greater
values has a rank of 3. The previous query returns all the rows with a rank of 5 or less, using the TOP clause.
However, it's not so simple to get a list of the rows with a rank of 5 or less along with an integer indicating the actual
rank value. Let's look at several possible solutions to this problem.

Approach 1: The Standard SQL Approach Using a View

First, let's look again at the titles table. Suppose you want to assign a rank value to all rows in the table based on
their ytd_sales values (with the highest value getting the top rank). You can do this by performing a ranking of the
values by nesting a SELECT inside the select list. For each row in the outer SELECT, the inner SELECT returns the
number of values in the table that are equal to or greater than the value being ranked. This "rows rank" value is then
correlated back to the main query. You can query the view to obtain any rank.

Performance suffers on large tables because a table must be scanned for every row. Ties are assigned equal rank;
unique numbers are not guaranteed unless the columns are known to be unique. In the next example, the next nontie
value's rank is one higher, not lower. Notice that we completely disregard rows with NULL values for ytd_sales.
CREATE VIEW ranked_sales (rank, title_id, ytd_sales, title)
AS
SELECT
(SELECT COUNT(DISTINCT T2.ytd_sales) FROM titles AS T2
WHERE T2.ytd_sales >= T1.ytd_sales ) AS rank,
title_id,
ytd_sales,
title
FROM titles AS T1 WHERE ytd_sales IS NOT NULL
GO

SELECT * FROM ranked_sales WHERE rank <= 10 ORDER BY rank

rank title_id ytd_sales title

---- -------- --------- ---------------------------------------------
1 MC3021 22246 The Gourmet Microwave
2 BU2075 18722 You Can Combat Computer Stress!
3 TC4203 15096 Fifty Years in Buckingham Palace Kitchens
4 PC1035 8780 But Is It User Friendly?
5 BU1032 4095 The Busy Executive's Database Guide
5 BU7832 4095 Straight Talk About Computers
5 PC8888 4095 Secrets of Silicon Valley
5 TC7777 4095 Sushi, Anyone?
6 PS3333 4072 Prolonged Data Deprivation: Four Case Studies
7 BU1111 3876 Cooking with Computers: Surreptitious Balance
Sheets
8 PS7777 3336 Emotional Security: A New Algorithm
9 PS2091 2045 Is Anger the Enemy?
10 MC2222 2032 Silicon Valley Gastronomic Treats

Approach 2: The Standard SQL Approach Without Using a View

This document is created with the unregistered version of CHM2PDF Pilot

This approach is basically the same as the first one, but it saves you the step of creating the view, at the possible cost
of being a bit harder to understand. Instead of using a view, you use a derived table (by adding a SELECT in the
FROM clause). Performance is identical to that of the view approach, however. (That is, it's not too good.) The
results are also identical, so I won't show them here.
SELECT rank, title_id, ytd_sales, title
FROM (SELECT
T1.title_id,
ytd_sales,
T1.title,
(SELECT COUNT(DISTINCT T2.ytd_sales) FROM titles AS T2
WHERE T1.ytd_sales <= T2.ytd_sales) AS rank
FROM titles AS T1) AS X
WHERE (ytd_sales IS NOT NULL) AND (rank <= 10)
ORDER BY rank

Approach 3: Using a Temporary Table with an identity Column and

No Ties
If you want to assign a unique number even in the case of ties, or if you know that ties will not occur, you can create
a temporary table with an identity column and then SELECT into the temporary table in an ordered fashion. This
gives you a materialized table with rankings. This approach is conceptually easy to understand, and it's fast. Its only
downside is that it does not recognize tied values. It's not ANSI-standard SQL, but this approach takes advantage of
SQL Server-specific features. This approach can be useful with other sequential operations, as you'll see later.
CREATE TABLE #ranked_order
(
rank int IDENTITY NOT NULL PRIMARY KEY,
title_id char(6) NOT NULL,
ytd_sales int NOT NULL,
title varchar(80) NOT NULL
)
GO

INSERT #ranked_order
SELECT title_id, ytd_sales, title FROM titles WHERE ytd_sales
IS NOT NULL ORDER BY ytd_sales DESC

SELECT * FROM #ranked_order

DROP TABLE #ranked_order

rank title_id ytd_sales title

---- -------- --------- ---------------------------------------------
1 MC3021 22246 The Gourmet Microwave
2 BU2075 18722 You Can Combat Computer Stress!
3 TC4203 15096 Fifty Years in Buckingham Palace Kitchens
4 PC1035 8780 But Is It User Friendly?
5 BU1032 4095 The Busy Executive's Database Guide
6 BU7832 4095 Straight Talk About Computers
7 PC8888 4095 Secrets of Silicon Valley
8 TC7777 4095 Sushi, Anyone?
9 PS3333 4072 Prolonged Data Deprivation: Four Case Studies
10 BU1111 3876 Cooking with Computers: Surreptitious Balance
Sheets
11 PS7777 3336 Emotional Security: A New Algorithm
12 PS2091 2045 Is Anger the Enemy?
13 MC2222 2032 Silicon Valley Gastronomic Treats
14 PS1372 375 Computer Phobic AND Non-Phobic Individuals
This document is created with the unregistered version of CHM2PDF Pilot

15 TC3218 375 Onions, Leeks, and Garlic: Cooking Secrets of

the Med
16 PS2106 111 Life Without Fear

Approach 4: Using a Temporary Table with an identity Column and

Ties
With standard SQL solutions, if the query focuses on four rows with two rows tied for second rank, then one row
has the rank of first, two rows are second, and one row is third. No row is ranked fourth. An alternative way to rank
them is to make one row first, two rows second, no row third, and one row fourth. (This is how standings in a race
are posted if two swimmers tie for second place.) After populating the temporary table, you can query the table for
the lowest rank for a given value and correlate that value back to the main query with a nested SELECT.
-- Approach 4A. Create a temp table with an identity, and then do
-- an ordered select to populate it.
-- Do a nested select correlated back to itself to find the lowest
-- rank for a given value.
CREATE TABLE #ranked_order
(
rank int IDENTITY NOT NULL,
title_id char(6) NOT NULL,
ytd_sales int NOT NULL,
title varchar(80) NOT NULL
)
GO

INSERT #ranked_order
SELECT title_id, ytd_sales, title FROM titles WHERE ytd_sales
IS NOT NULL ORDER BY ytd_sales DESC

SELECT B.rank, A.title_id, B.ytd_sales , A.title

FROM
(SELECT MIN(T2.rank) AS rank, T2.ytd_sales FROM #ranked_order AS T2
GROUP BY T2.ytd_sales) AS B,
#ranked_order AS A
WHERE A.ytd_sales=B.ytd_sales
ORDER BY B.rank

DROP TABLE #ranked_order

rank title_id ytd_sales title

---- -------- --------- ---------------------------------------------
1 MC3021 22246 The Gourmet Microwave
2 BU2075 18722 You Can Combat Computer Stress!
3 TC4203 15096 Fifty Years in Buckingham Palace Kitchens
4 PC1035 8780 But Is It User Friendly?
5 BU1032 4095 The Busy Executive's Database Guide
5 BU7832 4095 Straight Talk About Computers
5 PC8888 4095 Secrets of Silicon Valley
5 TC7777 4095 Sushi, Anyone?
9 PS3333 4072 Prolonged Data Deprivation: Four Case Studies
10 BU1111 3876 Cooking with Computers: Surreptitious Balance
Sheets
11 PS7777 3336 Emotional Security: A New Algorithm
12 PS2091 2045 Is Anger the Enemy?
13 MC2222 2032 Silicon Valley Gastronomic Treats
14 PS1372 375 Computer Phobic AND Non-Phobic Individuals
14 TC3218 375 Onions, Leeks, and Garlic: Cooking Secrets of
the Med
16 PS2106 111 Life Without Fear
This document is created with the unregistered version of CHM2PDF Pilot

You can slightly modify the SELECT on the temporary table and explicitly indicate where ties exist and how many
values were tied. The creation and population of the temporary table are identical, so I'll show just the SELECT
statement:
-- Approach 4B. Same as 4A, explicitly noting the ties.
CREATE TABLE #ranked_order
(
rank int IDENTITY NOT NULL,
title_id char(6) NOT NULL,
ytd_sales int NOT NULL,
title varchar(80) NOT NULL
)
GO

INSERT #ranked_order
SELECT title_id, ytd_sales, title FROM titles WHERE ytd_sales
IS NOT NULL ORDER BY ytd_sales DESC

SELECT B.rank,
CASE B.number_tied
WHEN 1 THEN ' '
ELSE '('+ CONVERT(varchar, number_tied) + ' Way Tie)'
END AS tie,
A.title_id,
B.ytd_sales,
A.title
FROM
(SELECT MIN(T2.rank) AS rank, COUNT(*) AS number_tied, T2.ytd_sales
FROM #ranked_order AS T2 GROUP BY T2.ytd_sales) AS B,
#ranked_order AS A
WHERE A.ytd_sales=B.ytd_sales
ORDER BY B.rank

DROP TABLE #ranked_order

rank tie title_id ytd_sales title
---- ----------- -------- --------- -----------------------------------------
1 MC3021 22246 The Gourmet Microwave
2 BU2075 18722 You Can Combat Computer Stress!
3 TC4203 15096 Fifty Years in Buckingham Palace Kitchens
4 PC1035 8780 But Is It User Friendly?
5 (4 Way Tie) BU1032 4095 The Busy Executive's Database Guide
5 (4 Way Tie) BU7832 4095 Straight Talk About Computers
5 (4 Way Tie) PC8888 4095 Secrets of Silicon Valley
5 (4 Way Tie) TC7777 4095 Sushi, Anyone?
9 PS3333 4072 Prolonged Data Deprivation
10 BU1111 3876 Cooking with Computers
11 PS7777 3336 Emotional Security: A New Algorithm
12 PS2091 2045 Is Anger the Enemy?
13 MC2222 2032 Silicon Valley Gastronomic Treats
14 (2 Way Tie) PS1372 375 Computer Phobic AND Non-Phobic Individuals
14 (2 Way Tie) TC3218 375 Onions, Leeks, and Garlic
16 PS2106 111 Life Without Fear

Finding Differences Between Intervals

Suppose we have a table with two columns that records data for measured temperatures. The first column is the
datetime at which the measurement was taken, and the second column is the temperature. We want to find the
change in temperature between one interval and the next, and we want it expressed as the absolute value of the
number of degrees that changed per minute. There is no preexisting primary key, such as measurement_id. It's
unlikely that duplicate measurements exist for any single time (that is, the datetime field is probably unique), but we
This document is created with the unregistered version of CHM2PDF Pilot

don't know this for sure. And the intervals between data measurements are not consistent. In fact, the data was not
inserted in order. Like the rankings problem, this problem can be solved using standard (but tricky) SQL. In fact, you
can think of this as another type of ranking problem—but here, you want to see the differences between adjacent
ranks.

First, I'll set up the table with some fairly random data. I really don't care whether the temperature values are realistic
or how much they might fluctuate in even a few minutes. So if you run this example and see temperatures of -100º F
or a 40-degree change in temperature in five minutes, don't worry. These readings might not be from Earth!
NOTE
It's worthwhile to set up the routine to generate test data even though in this case it would be quicker to manually
insert 20 rows of data. Once the routine is working and you've checked your solution on the small table, you can
easily change the constant in the WHILE loop to add much more data. This is important because many solutions
perform well with small amounts of data but can degrade badly with large amounts of data. This can be especially true
if a solution uses correlated subqueries or self-joins or if it does table scans.
-- diff_intervals.sql
IF NULLIF(OBJECT_ID('measurements'), 0) > 0
DROP TABLE measurements
GO

CREATE TABLE measurements

(
when_taken datetime NOT NULL,
temperature numeric(4, 1) -- (Fahrenheit)
)
CREATE CLUSTERED INDEX measurements_idx01
ON measurements (when_taken)
GO

DECLARE @counter int, @whendate datetime, @val numeric(4, 1),

@randdiff smallint, @randmins smallint
SELECT @counter=1, @whendate=GETDATE(), @val=50.0
/* Insert 20 rows of data. Change constant if you want more. */
WHILE (@counter <= 20)
BEGIN
INSERT measurements VALUES (@whendate, @val)
-- Get a random number between -20 and 20 for change in
-- temperature. This will be added to the previous value,
-- plus RAND again to give a fractional component.
SELECT
@randdiff=CASE
WHEN CONVERT(int, RAND() * 100) % 2 = 1 THEN
CONVERT(int, RAND() * 1000) % 21 * -1
ELSE CONVERT(int, RAND() * 1000) % 21
END,
-- Get a random number between 0 and 10080 (the number of minutes
-- in a week). This will be added to the current GETDATE
-- value. Since GETDATE returns a value to the millisecond,
-- it's unlikely there will ever be a duplicate, although it
-- is possible if the result of the addition and the current
-- GETDATE() value happen to collide with the addition in
-- another row. (That's intentional; we are not assuming
-- that duplicates are automatically prevented.)
@randmins=CONVERT(int, RAND() * 100000) % 10080
SELECT @counter=@counter + 1,
@whendate=DATEADD(mi, @randmins, GETDATE()),
@val=@val + @randdiff + RAND()
END

SELECT * FROM measurements

when_taken temperature
This document is created with the unregistered version of CHM2PDF Pilot

--------------------------- -----------
2000-07-17 12:31:41.203 50.0
2000-07-17 16:07:41.243 48.0
2000-07-17 23:56:41.233 55.7
2000-07-18 00:13:41.243 32.8
2000-07-18 13:40:41.273 -18.2
2000-07-18 20:35:41.283 27.2
2000-07-18 21:53:41.243 34.6
2000-07-19 01:21:41.283 -34.6
2000-07-19 22:52:41.253 16.8
2000-07-20 03:56:41.283 -27.8
2000-07-20 06:56:41.283 38.1
2000-07-21 08:28:41.243 67.6
2000-07-21 20:43:41.283 -7.0
2000-07-22 19:12:41.293 20.9
2000-07-23 05:59:41.273 .4
2000-07-23 07:33:41.243 58.1
2000-07-23 09:25:41.243 65.6
2000-07-23 16:04:41.283 7.0
2000-07-23 16:05:41.253 2.2
2000-07-23 20:59:41.253 30.9

Approach 1: Using Standard SQL

This approach is similar to the rankings solution. I assign a ranking to each row and then join the table back to itself
on the ranking value less 1. If duplicate datetime values exist, this approach still computes the differential to the
previous (nonduplicate) measurement, which might or might not be how you'd want to deal with it. As with the
rankings solution, you can use this approach either as a view or as a derived table. Since a view is easier for most
people to understand, I'll write it that way.

As in the top n and rankings problems, this approach is a good brainteaser, but it's not a good performer if the table
is anything but very small. In addition, this approach is problematic if duplicate datetime values might exist.
CREATE VIEW rankdates (when_taken, temperature, daterank)
AS
SELECT when_taken, temperature,
(SELECT COUNT(DISTINCT when_taken) FROM measurements AS T1
WHERE T1.when_taken <= T0.when_taken) AS rank
FROM measurements AS T0
GO
SELECT * FROM rankdates ORDER BY daterank
GO

when_taken temperature daterank

--------------------------- ----------- --------
2000-07-17 12:31:41.203 50.0 1
2000-07-17 16:07:41.243 48.0 2
2000-07-17 23:56:41.233 55.7 3
2000-07-18 00:13:41.243 32.8 4
2000-07-18 13:40:41.273 -18.2 5
2000-07-18 20:35:41.283 27.2 6
2000-07-18 21:53:41.243 34.6 7
2000-07-19 01:21:41.283 -34.6 8
2000-07-19 22:52:41.253 16.8 9
2000-07-20 03:56:41.283 -27.8 10
2000-07-20 06:56:41.283 38.1 11
2000-07-21 08:28:41.243 67.6 12
2000-07-21 20:43:41.283 -7.0 13
2000-07-22 19:12:41.293 20.9 14
2000-07-23 05:59:41.273 .4 15
2000-07-23 07:33:41.243 58.1 16
This document is created with the unregistered version of CHM2PDF Pilot

2000-07-23 09:25:41.243 65.6 17

2000-07-23 16:04:41.283 7.0 18
2000-07-23 16:05:41.253 2.2 19
2000-07-23 20:59:41.253 30.9 20

-- Correlate each value with the one right before it

SELECT
P1_WHEN=V1.when_taken, P2_WHEN=V2.when_taken,
P1=V1.temperature, P2=V2.temperature,
DIFF=(V2.temperature - V1.temperature)
FROM rankdates AS V1 LEFT OUTER JOIN rankdates AS V2
ON (V2.daterank=V1.daterank + 1)
GO

P1_WHEN P2_WHEN P1 P2 DIFF

----------------------- ----------------------- ----- ----- ----
2000-07-17 12:31:41.203 2000-07-17 16:07:41.243 50.0 48.0 -2.0
2000-07-17 16:07:41.243 2000-07-17 23:56:41.233 48.0 55.7 7.7
2000-07-17 23:56:41.233 2000-07-18 00:13:41.243 55.7 32.8 -22.9
2000-07-18 00:13:41.243 2000-07-18 13:40:41.273 32.8 -18.2 -51.0
2000-07-18 13:40:41.273 2000-07-18 20:35:41.283 -18.2 27.2 45.4
2000-07-18 20:35:41.283 2000-07-18 21:53:41.243 27.2 34.6 7.4
2000-07-18 21:53:41.243 2000-07-19 01:21:41.283 34.6 -34.6 -69.2
2000-07-19 01:21:41.283 2000-07-19 22:52:41.253 -34.6 16.8 51.4
2000-07-19 22:52:41.253 2000-07-20 03:56:41.283 16.8 -27.8 -44.6
2000-07-20 03:56:41.283 2000-07-20 06:56:41.283 -27.8 38.1 65.9
2000-07-20 06:56:41.283 2000-07-21 08:28:41.243 38.1 67.6 29.5
2000-07-21 08:28:41.243 2000-07-21 20:43:41.283 67.6 -7.0 -74.6
2000-07-21 20:43:41.283 2000-07-22 19:12:41.293 -7.0 20.9 27.9
2000-07-22 19:12:41.293 2000-07-23 05:59:41.273 20.9 .4 -20.5
2000-07-23 05:59:41.273 2000-07-23 07:33:41.243 .4 58.1 57.7
2000-07-23 07:33:41.243 2000-07-23 09:25:41.243 58.1 65.6 7.5
2000-07-23 09:25:41.243 2000-07-23 16:04:41.283 65.6 7.0 -58.6
2000-07-23 16:04:41.283 2000-07-23 16:05:41.253 7.0 2.2 -4.8
2000-07-23 16:05:41.253 2000-07-23 20:59:41.253 2.2 30.9 28.7
2000-07-23 20:59:41.253 NULL 30.9 NULL NULL

NOTE
If you're interested in the derived table solution, go back to the rankings problem and note the difference between a
view and a derived table; it should be easy to see how the derived table solution can be done here as well.

Approach 2: Materializing Rankings and Using a Self-Join

Conceptually, this solution is similar to the standard SQL approach. But rather than using a true view, we'll
materialize the rankings as a temporary table, similar to the approach in one of the rankings solutions. Then we'll do
the self-join with the temporary table. The advantage, of course, is that we create the rankings only once instead of
many times. This is faster because the identity value simply assigns the rank in the order that the rows are presented.

In SQL Server 2000, this solution performs about 33 percent faster than the standard SQL solution. The results are
the same, so I won't bother repeating the output. The standard SQL approach takes 122 logical reads with this
dataset; the temporary table approach here takes 44 logical reads to create the temp table and 42 logical reads to
access it. Because so little data is involved, no more than two physical reads occur in either of these solutions.
CREATE TABLE #rankdates (
when_taken datetime,
temperature numeric(4, 1),
daterank int IDENTITY PRIMARY KEY)
GO
INSERT #rankdates (when_taken, temperature)
SELECT when_taken, temperature
FROM measurements
This document is created with the unregistered version of CHM2PDF Pilot

ORDER BY when_taken ASC

GO

SELECT
P1_WHEN=V1.when_taken, P2_WHEN=V2.when_taken,
P1=V1.temperature, P2=V2.temperature,
DIFF=(V2.temperature - V1.temperature)
FROM #rankdates AS V1 LEFT OUTER JOIN #rankdates AS V2
ON (V2.daterank=V1.daterank + 1)
GO

DROP TABLE #rankdates

Selecting Instead of Iterating

One common need is to perform some operation for every value, from 1 to n. Earlier in this chapter, I used the
trigonometry functions to produce a listing of sine/cosine/tangent for every 10 degrees between 0 and 180. I took the
obvious route and wrote a simple loop. However, that approach results in every row being its own result set. As I've
mentioned many times, sending many result sets when one will do is inefficient, and it is also more cumbersome to
process in the client application. To refresh your memory, here's the solution I used earlier:
DECLARE @degrees smallint
DECLARE @radians float
SELECT @degrees=0
SELECT @radians=0
WHILE (@degrees <= 180)
BEGIN
SELECT
DEGREES=@degrees,
RADIANS=STR(@radians, 8, 5),
SINE=STR(SIN(@radians), 8, 5),
COSINE=STR(COS(@radians), 8, 5),
TANGENT=STR(TAN(@radians), 8, 5)
SELECT @degrees=@degrees + 10
SELECT @radians=RADIANS(CONVERT(float, @degrees))
END

Since it's common to iterate in this way, a simple table of ordered numbers can come in handy:
-- Create the seq_num table
CREATE TABLE seq_num
(seq_num INT PRIMARY KEY NOT NULL)

-- Populate the seq_num table with values from -500 through 500
DECLARE @counter int
SELECT @counter= -500
WHILE (@counter <= 500)
BEGIN
INSERT seq_num VALUES (@counter)
SELECT @counter=@counter + 1
END

-- If doing this for real, you might as well set FILLFACTOR to 100

I can select or join to this handy seq_num table rather than iterating. The following solution runs much faster than the
earlier solution. It returns a single result set, and it's easier to write:
SELECT
This document is created with the unregistered version of CHM2PDF Pilot

ANGLE=seq_num * 10,
SINE=STR(SIN(seq_num * 10), 7, 4),
COSINE=STR(COS(seq_num * 10), 7, 4),
TANGENT=STR(TAN(seq_num * 10), 7, 4)
FROM seq_num
WHERE seq_num BETWEEN 0 AND 18

ANGLE SINE COSINE TANGENT

----- ------- ------- -------
0 0.0000 1.0000 0.0000
10 -0.5440 -0.8391 0.6484
20 0.9129 0.4081 2.2372
30 -0.9880 0.1543 -6.4053
40 0.7451 -0.6669 -1.1172
50 -0.2624 0.9650 -0.2719
60 -0.3048 -0.9524 0.3200
70 0.7739 0.6333 1.2220
80 -0.9939 -0.1104 9.0037
90 0.8940 -0.4481 -1.9952
100 -0.5064 0.8623 -0.5872
110 -0.0442 -0.9990 0.0443
120 0.5806 0.8142 0.7131
130 -0.9301 -0.3673 2.5323
140 0.9802 -0.1978 -4.9554
150 -0.7149 0.6993 -1.0223
160 0.2194 -0.9756 -0.2249
170 0.3466 0.9380 0.3696
180 -0.8012 -0.5985 1.3387
This document is created with the unregistered version of CHM2PDF Pilot

34

Full-Text Searching
The Transact-SQL language in SQL Server 2000 includes several programming extensions that allow you to access
SQL Server data that is maintained and accessed by the Microsoft Search service. These extensions allow you to use
special indexes to perform powerful, flexible searches that are beyond the capacity of normal SQL statements. The
searching that you can do with SQL Server full-text searching is similar to the searching you can do in SQL Server
Books Online or online resources such as MSDN and TechNet, where you supply any combination of keywords and
the application returns a list of "hits," ranked in order by how closely they match your keyword list. Although these
tools do not use the Microsoft Search service, they can give you an idea of the type of searching that is possible. For
an example of a real site using Microsoft full-text search technology, you can take a look at the MSN E-Shop at
www.eshop.com.

You're aware by now that you can do searches with normal SQL, using a LIKE operator. For example, we can use
the following query to find all the employees in the Northwind database who mention French in their notes field:
USE Northwind
SELECT LastName, FirstName, Notes
FROM EMPLOYEES
WHERE notes like '%french%'

This should return five rows. But what if we then want to find all the employees who list both German and French?
We could try executing the following query:
SELECT LastName, FirstName, Notes
FROM EMPLOYEES
WHERE notes like '%german%french%'

However, this query does not return any rows because the pattern matching of LIKE looks for parts of the string in
the order listed: SQL Server returns only rows in which the notes field contains some (or no) characters, followed by
the string german followed by some (or no) characters, followed by the string french followed by some (or no)
characters. If the string french comes before the string german, SQL Server does not return the row. However, this
query returns two rows:
SELECT LastName, FirstName, Notes
FROM EMPLOYEES
WHERE notes like '%french%german%'

Using the full-text search capability, we could write a similar query to the above and list German and French in any
order. Of course, we could write this query with two separate conditions in order to find either order of french and
german. However, having the flexibility to write our query with the search strings in any order is only one of the
advantages of using SQL Server's full-text search features.

The biggest advantage of using the full-text search functionality is that each word in your textual data is indexed, so
queries looking for parts of strings can be amazingly fast. When using its own internal indexes, SQL Server considers
all the data stored in the key field as a single value. Consider an index on the notes field in the Northwind database's
employees table. The index would help us quickly find how many notes fields start with the word I but would be of
no use at all in finding which rows have the word french in their notes field because the entire notes field is sorted
starting with the first character. A full-text index, on the other hand, could be a big help because it keeps track of each
word in the notes field independently.
This document is created with the unregistered version of CHM2PDF Pilot

The full-text searching feature in SQL Server 2000 actually consists of two basic components: full-text indexing,
which lets you create and populate full-text catalogs that are maintained outside of SQL Server and managed by the
Microsoft Search service, and full-text searching, which uses four new Transact-SQL operations—CONTAINS,
FREETEXT, CONTAINSTABLE, and FREETEXTTABLE). The CONTAINS and FREETEXT operators are
predicates used in a WHERE clause, and CONTAINSTABLE and FREETEXTTABLE are rowset functions used in
a FROM clause. These operations allow you to query your full-text catalogs.

Full-Text Indexes
SQL Server 2000 allows you to create full-text indexes with support from Microsoft Search service. The Search
service creates and manages full-text indexes and can work in conjunction with other products in addition to SQL
Server 2000, such as Windows 2000 and Microsoft Exchange Server 2000. The indexing components of the Search
service use a word-breaking algorithm to take specified columns of textual data and break them into individual words.
These words are then stored in a full-text index, which is external to SQL Server's data storage.

Figure 10-4 shows the relationship of the Search service to SQL Server's relational and storage engines. The SQL
Server handler is a driver that contains the logic to extract the textual data from the appropriate SQL Server columns
and pass them to the indexing components of the Search service. The handler can also pass values from the Search
service to the SQL engine to indicate how to repopulate the indexes. I'll tell you about population techniques shortly.

Figure 10-4. Full-text indexing support, reproduced from SQL Server documentation.

In the most general sense, you can think of full-text indexes as similar to SQL Server indexes in that they organize
your data so you can find information very quickly. But the similarity stops there. I'll list the most important differences
here and elaborate on most of them in the sections that follow.
 SQL Server indexes are stored in your database files; full-text indexes are stored externally in operating
system files.
 You can have several (actually, up to 250) SQL Server indexes per table; you can only have one full-text
index per table.
 There is only one entry in each SQL Server index for each row of data; there can be many entries in a full-text
index for each row because each row is broken into its separate words.
 SQL Server indexes are maintained individually; full-text indexes are grouped together in catalogs for ease of
maintenance.
 SQL Server indexes are automatically updated as you modify your data; full-text indexes are updated
asynchronously. This is called repopulation of full-text indexes and you can schedule repopulation to occur at
This document is created with the unregistered version of CHM2PDF Pilot

regular times, or you can manually request a repopulation to take place.

Setting Up Full-Text Indexes

You need only take a few basic steps to get your data set up for full text searching. An important prerequisite is to
make sure that the full-text indexing feature has been installed with your SQL Server. In SQL Server 7, the tools for
full-text indexing were not installed by default; if you've upgraded a SQL Server 7 server on which the fulltext indexing
was not installed, your SQL Server 2000 server also will not install full-text indexing. A new installation of SQL
Server 2000 will have full-text indexing installed by default if you choose the Typical installation option. (One
exception is if you are installing SQL Server 2000 on Windows 2000 Professional or Windows NT 4 Workstation; in
these cases you must choose a custom installation in order to have full-text indexing installed.) To choose not to install
full-text indexing, you have to choose the Custom installation and uncheck the box for full-text indexing. To install
full-text indexing after installation, you can rerun the setup program from your SQL Server 2000 CD and choosing the
option to add new components to an existing installation.
NOTE
There are several ways to determine whether full-text indexing has been installed for your SQL Server. The easiest
way is probably to expand any user database in the left pane in SQL Server Enterprise Manager and see if you have
a folder called Full-Text Catalogs. Alternatively, you can examine use the DATABASEPROPERTYEX metadata
function, and check the value of the option 'IsFullTextEnabled'.

You can set up full-text indexing through wizards and dialog boxes in SQL Server Enterprise Manger or by using
stored procedures. (The full-text wizard is not available if you haven't installed full-text indexing). In this section, I'll
describe the general process of setting up full-text searching using system stored procedures. Once you understand
what's involved, the GUI should be completely self-explanatory. I'll show examples of most of the stored procedures,
which of course are called behind the scenes by SQL Server Enterprise Manager. The examples that follow set up
full-text indexing on the products table in the Northwind database. If you execute these examples for setting up
full-text indexing, you'll be able to run the full-text querying examples later in this section.

The first step in setting up full-text indexing is to enable a database to allow this feature. From the command line in
SQL Query Analyzer, you can run this procedure:
USE Northwind
EXEC sp_fulltext_database 'enable'

Note that the name of the database is not an argument in the procedure; this procedure enables the current database
for full-text indexing and searching.
WARNING
You should enable a database for full-text indexing only once. If you run this procedure a second time, any catalogs
that you have already created will have to be completely repopulated.

The second step is to create a catalog in the operating system to store the full-text indexes. A catalog can contain the
indexes from one or more tables, and it is treated as a unit for maintenance purposes. In general, you shouldn't need
more than one catalog per database. You may find that a huge table of more than 10 million rows will benefit from
being in its own catalog, but it would really depend on how often that table is being updated and how often the
full-text index needs repopulating.

From SQL Query Analyzer, you can build a catalog that specifies a full path for the directory in which to store the
catalog, as shown here:
EXEC sp_fulltext_catalog 'Northwind_data', 'create', 'D:\FTcatalogs'

Specifying the directory path is optional, and the default is NULL. A NULL for the path indicates that the default
This document is created with the unregistered version of CHM2PDF Pilot

location should be used. This is the FTDATA subdirectory in the install directory for your SQL Server instance; for
example, C:\Program Files\Microsoft SQL Server\MSSQL$<instance name>\Ftdata. The specified root directory
must reside on a drive on the same computer as your SQL Server instance. The drive on which you're building the
catalog must have at least 50 MB of free space; otherwise, you'll get an error and will have to free up space or
choose another location. If you're using the full-text index wizard, you will actually specify the tables and columns first,
and then choose whether to store the full-text indexes in a new or existing catalog.

Any table that will have a full-text index must already have a unique index. If the table has more than one unique
index, you will need to choose one of them to be the index used by full-text indexing. In the full-text index, the
individual words are stored along with an indication of where in the original table those words can be found. The
location is indicated by a key value for the unique index. For example, suppose we have a full-text index on the notes
field in the titles table in the pubs database. The unique key for the titles table is the title_id column. A subset of the
full-text index might look something like this:
business BU1032
computer: BU7832 PC8888 PC9999 PS1372 PS7777
software PC1035 PC8888

Each keyword is followed by a list of all the title_id values that have the indicated keyword in the row for that
title_id. Note that this is just an abstraction of the way the full-text index would really look. The individual keys are
actually encoded before they are stored, and the keys are found by using a hashing algorithm. Also, each keyword
stores more than just a unique index key; it also stores information about what table the key is in and the position
within the row where the word can be found.

To specify which tables to index, which column will be the full-text index key, and what catalog the index should be
stored in, you can use the following procedure:
EXEC sp_fulltext_table 'products', 'create', 'Northwind_data',
'PK_Products'

As I mentioned, you can have at most one full-text index on a table, but any number of columns can be included in
that index. The columns must be character based, or of type image: char, varchar, nchar, nvarchar, text, imaget
or ntext. The columns can be different datatypes, for example, there can be one char column and one text column,
and the columns can even be based on different languages. Remember that the language used to determine the
word-breaking algorithm and the inflectional forms allowed for words in a particular column does not have to
explicitly defined. The default is determined by the configuration option 'default full-text language'. On my SQL
Server, this option has the value 1033, which is equivalent to 0x0409. The Books Online description for the
sp_fulltext_column procedure gives you the locale ID values for all the languages supported for full-text indexing,
and you can see that 0x0409 is the ID for US English. When defining a column to add to a full-text index, the fourth
parameter is the locale ID. Leaving this value out or supplying an explicit NULL means that the default locale ID will
be used. You can specify a 0 to mean that neutral word-breaking and inflectional algorithms should be used, for
example, if the column contains data in multiple languages or in an unsupported language.

If using stored procedures, each column must be added separately:

EXEC sp_fulltext_column 'products', 'ProductName', 'add', 0
EXEC sp_fulltext_column 'products', 'QuantityPerUnit', 'add', 0

That's all that you need to do to create full-text indexes. However, the information that the index stores is not
collected automatically. You must manually maintain the indexes by determining when they should be populated.

Maintaining Full-Text Indexes

This document is created with the unregistered version of CHM2PDF Pilot

Populating a full-text index simply means building it. The process of building a full-text index is quite different from
creating a SQL Server index because the external Search service is involved. To create the original index, you do a
full population, and SQL Server takes every row from the indexed table and passes it to the Search service. The
Search service applies an algorithm called word-breaking to each row of each indexed column in order to generate
the individual words that will be part of the index. The word-breaking algorithm is unique for each supported language
and determines what is considered the end of one word and the beginning of another. For example, not all languages
break words with spaces. The full-text index keeps track of each word in each field, what unique keys each word is
associated with, and where in each column the words appears. When I tell you about querying your full-text indexes,
you'll see that you can ask for words that are "near" other words, and in order to determine "nearness," the full-text
index must know whether a word occurs as the 2nd word in a column or the 222nd word.

The index-building procedure ignores certain words that are considered unimportant. These are called noise words,
and the list is different for each language. There is a predefined list of noise words for each language supported, and
these lists are stored as ASCII files that you can edit using any text editor. The noise words typically include
pronouns, prepositions, conjunctions, numbers, and individual letters. You can take a look at the noise word files to
see what else is ignored. You can add words to be ignored or remove words from the list that you want to have in
your full-text indexes. The Search service has its own set of noise word files that are separate from SQL Server noise
word files. If there is no SQL Server file for a particular language, the one for the Search service is used. The Search
service noise word files are stored on the partition with your operating system files, in the directory \Program
Files\Common Files\System\MSSearch\Data\Config.

The SQL Server full-text index noise words files are in the same directory that your catalogs are stored in, which by
default is \Program Files\Microsoft SQL Server\MSSQL$<instance name>\Ftdata. The noise word files are in a
subdirectory under Ftdata called SQLServer$<instance name>\Config. The file containing the US English language
noise words is called noise.enu, the file containing the UK English noise words is noise.eng, the file containing the
French noise words is noise.fra, and the Spanish file is noise.esn. Each language has its own suffix.

Full Population of Full-Text Indexes

Full population can take a long time for a large table or a catalog—up to many hours for a catalog involving a million
rows. Obviously, you won't want to have to do a full population more often than is necessary. However, if most of
your data is changing on a regular basis—for example, if 90 percent of your keys are changing every day—you might
find it necessary to fully repopulate your full-text index. From SQL Query Analyzer, you can start a full population
with this command:
EXEC sp_fulltext_catalog 'Northwind_data', 'start_full'

Note that populating or repopulating is done on a catalog, whether the catalog contains data from one table or from
several. All the words from all the tables in the catalog are stored together, and the index keeps track of which table
each word came from.

The population is done asynchronously, and the preceding command only starts the process. The property function
FULLTEXTCATALOGPROPERTY tells you the current status of a full-text catalog, including whether the full
population has finished. This function also tells you the number of keys for all indexes in the catalog and the elapsed
time since the last population. To find out if a full population has completed, you can execute the following:
SELECT FULLTEXTCATALOGPROPERTY ( 'Northwind_data', 'PopulateStatus' )

A return value of 0 means there is no current population activity, a value of 1 means a full population is in progress,
and a value of 6 means an incremental population is in progress. You can use the
FULLTEXTCATALOGPROPERTY function and look for the value of the property PopulateCompletionAge to see
when the population finished. The value is returned in number of seconds since a base date of January 1, 1990. We
can then use the SQL Server dateadd function to find out the actual date that is being referenced:
This document is created with the unregistered version of CHM2PDF Pilot

SELECT dateadd(ss, FULLTEXTCATALOGPROPERTY ( 'Northwind_data',

'PopulateCompletionAge' ), '1/1/1990' )

NOTE
Keep in mind that if a property function returns NULL, it means you've typed something in incorrectly or the function
call has failed for some reason. The FULLTEXTCATALOGPROPERTY function returns 0 if no activity is in
progress. Since this is an asynchronous activity, you might need to pause while you wait for the population to
complete. You can check the status of the population in a loop, but since the population could take hours, I
recommend that you avoid continuously checking and looping. You should probably pause between checks. For a full
population, you might want to check the status every 10 minutes, and for an incremental population, you might want to
check every 2 minutes. A simple loop looks like this:
WHILE FULLTEXTCATALOGPROPERTY ( 'Northwind_data',
'PopulateStatus' ) = 1
WAITFOR DELAY '0:10:00' -- pause for 10 minutes
-- Now you can use your full-text index!

You have two alternatives to full population to keep your full-text indexes up to date: incremental population and
change tracking. Both require that you first do a full population of your catalog.

Incremental Population
Incremental population affects only rows that have changed since the last full or incremental population. When you
start an incremental population, the Search service sends a request to SQL Server, along with the timestamp of the
last population, and SQL Server returns all the rows that have changed. Only those rows are processed, and only the
words in those rows are used to update the full-text index. If a table does not have a timestamp column, an
incremental population will automatically revert to a full population with no warning. Since a full population can take
many times longer than an incremental population, you should be very sure that your tables have a timestamp field
before you request an incremental population.

You can start an incremental population manually using this command:

EXEC sp_fulltext_catalog 'Northwind_data', 'start_incremental'

In addition, you can use SQL Server Agent to schedule population of the full-text indexes at whatever interval you
need. Remember that no automatic updating of full-text indexes is done; if new data has been added to a table with a
full-text index, your full-text search queries won't find that data until the full-text index has been updated. In SQL
Server Enterprise Manager, you can right-click the name of the catalog or the table in the right pane and choose to
start a population or create a schedule for a population.

Change Tracking
You can also have changes automatically tracked in a table with a full-text index. You can do this in SQL Server
Enterprise Manager by right-clicking on the table name in the right pane after expanding the Tables folder. Note that
you don't enable change tracking for a catalog, only for an individual table. From SQL Query Analyzer, you can use
this command:
EXEC sp_fulltext_table 'products', 'Start_change_tracking'

If no full population has been done, SQL Server will do a full population when this procedure is called. If the full
population is in progress already, calling this procedure will restart the full population. If a full population has already
been done, an incremental update will be done so that when change tracking begins, the full-text index is known to be
This document is created with the unregistered version of CHM2PDF Pilot

completely up-to-date. Once change tracking is enabled, the system table sysfulltextnotify is updated every time any
of the columns in the full-text index is modified. This table basically just keeps track of the table ID and the value of
the unique index key for the rows that have been modified.

You can apply these tracked changes in two ways. This procedure updates all the appropriate full-text index rows
based on the changes in the sysfulltextnotify table:
EXEC sp_fulltext_table 'products', 'Update_index'

As with incremental population, you can schedule this update at whatever interval you choose. Alternatively, you can
choose to have the updating of the full-text index be an ongoing activity, using this command:
EXEC sp_fulltext_table 'products', 'Start_background_updateindex'

Once the background updating has started, the maintenance of the full-text index will be close to real time.
Depending on the load on your system, there might still be a lag between the time you update the table and the time
the modified data shows up in the full-text index, but in many cases the change to the index will seem to be immediate.

I recommend that you use change tracking whenever possible. If most of the data in the table is changing on a regular
basis, you might need to do repeated full populations. But as an alternative to incremental population, change tracking
is a clear performance winner. Two aspects of incremental population make it very time consuming. First, the
timestamp in a row can tell you only that an existing row has changed or that it is a new row. Inserts and updates can
be handled by this mechanism, but what happens if a row is deleted? How can the Search service know to remove an
entry from the full-text index? After all the rows in the base table are compared to the previous timestamp in order to
check for new or updated rows, the Search service compares all keys in the full-text index with all keys in the base
table to determine which ones might have been deleted. Admittedly, this is not nearly as time consuming as full
population, in which each word in each column must be isolated, but it can take quite a long time for a large table. The
second shortcoming of incremental population is that the timestamp is changed when any change is made to a row,
even if the change did not involve any of the full-text indexed columns. So if you update all the prices in the products
table, the timestamp in every row will change. When it comes time to do the incremental population, SQL Server will
send every single row to the Search service to be completely reindexed. This activity could take about as long as the
initial population and is very undesirable.

The use of change tracking avoids both of these problems. The sysfulltextnotify table gets a simple entry only when
a change is made to a full-text indexed column, and each row contains only three values: the ID of the table that was
changed, the key value corresponding to the changed row, and a field indicating the operation that took place (insert,
update, or delete). This simple structure contains all the information that is needed to make the necessary changes to
update the full-text index.

Querying Full-Text Indexes

As I mentioned earlier, there are four predicates you can use to search data that has been full-text indexed. I won't
go into great detail on the nuances of these predicates because the SQL Server 2000 documentation, as well as some
of the SQL Server programming books listed in the bibliography, adequately illustrate the types of queries you can
run. I'll just give you some of the basics and some simple examples.

To see some interesting results, you might need some data beyond what comes with the pubs or Northwind sample
databases. On the companion CD I put a script named ModifyProductNames.sql that modifies the data in the
products table in the Northwind database, and it illustrates some of the basic full-text searching techniques. The
products table contains the names of all the products sold by the Northwind Traders Company, and I modified these
product names by adding "Hot and Spicy," "Spicy Hot," "Extra Hot," and "Supreme" at the beginning or end of
various product names. This script contains five UPDATE statements to make the changes, but first it increases the
size of the ProductName column to allow for these new names. The sample queries below will run against a
This document is created with the unregistered version of CHM2PDF Pilot

Northwind database that includes these changes.

CONTAINS and CONTAINSTABLE Predicates

The most general purpose of the four full-text predicates is CONTAINS. It takes two arguments: the first is the
column you want to search in the table, and the second is the word or words you want to search for. Single quotes
enclose the complete search value that will be sent to the Search service, and that search value can contain multiple
words as well as the conjunctions AND, OR, and NOT. You can also use the conjunctions & (for AND), | (for OR)
and || (for NOT). You can use the wildcard character (*) only at the end of a search word. Here are a few examples:
--USE Northwind
-- This query returns 25 rows
SELECT * FROM products
WHERE CONTAINS(*, 'hot')

-- This query returns rows that have both hot and spicy anywhere
-- in the row: 24 rows
SELECT * FROM products
WHERE CONTAINS(*, 'hot and spicy')

-- This query returns rows that have hot but not spicy: 1 row
SELECT * FROM products
WHERE CONTAINS(*, 'hot and not spicy')

-- This query returns rows that contain any words that start with
-- "ch." Note that the double quotes are needed; otherwise,
-- the "*" will not be interpreted as a wildcard. The result set is
-- 9 rows, containing products that include "chocolate," "chowder,"
-- "chai," and "chinois."
SELECT * FROM products
WHERE CONTAINS(*, '"ch*"')

-- This query returns rows that contain the string "hot and spicy";
-- note the double quotes inside the single quotes:
-- 15 rows
SELECT * FROM products
WHERE CONTAINS(*, '"hot and spicy"')

The last query is different from one that has this condition: WHERE CONTAINS(*, 'hot and spicy'). If SQL Server
sends only the string hot and spicy to the Search service, the Search service will recognize it as two words conjoined
by the AND operator. However, when the whole string is in double quotes, it becomes one search string and the
AND is no longer an operator, it is part of the search string. The problem is that the Search service doesn't keep
track of the word and because it is a noise word, so it has no idea which rows in the products table have hot and
spicy. All it can do is find rows that have hot separated by one noise word from spicy. So the following query will
return exactly the same rows:
SELECT * FROM products
WHERE CONTAINS(*, '"hot or spicy"')

The following query will not return any rows because no rows have hot right next to spicy, in that order:
SELECT * FROM products
WHERE CONTAINS(*, '"hot spicy"')

And this query will also return no rows. Sweet is not a noise word, so it must actually appear in the data, and it
doesn't—at least not between hot and spicy.
This document is created with the unregistered version of CHM2PDF Pilot

SELECT * FROM products

WHERE CONTAINS(*, '"hot sweet spicy"')

The CONTAINS predicate can also look for inflectional forms of words—the plural, singular, gender, and neutral
forms of nouns, verbs, and adjectives will match the search criteria. The data in the products table doesn't give us too
many inflectional forms of the same word, but here's one example:
-- Find all rows containing any form of the word "season."
-- One row, with "seasoning," will be returned.
SELECT * FROM products
WHERE CONTAINS(*, 'FORMSOF( INFLECTIONAL,season)' )

Note that the syntax is quite awkward here because you need the nested predicate FORMSOF with the argument of
INFLECTIONAL. Remember that everything inside the single quotes is sent to the Search service, which requires
this syntax. In future versions of the product, you'll be able to expand FORMSOF to look for other forms besides
inflectional—for example, synonyms (THESAURUS), words that might be misspelled, or words that sound the same.

You can use full-text searching to look for words that appear near each other, using either the word NEAR or the
tilde (~). The meaning of near is imprecise by design, which allows the Search service to return rows containing
words of various degrees of nearness. NEAR has to take into account numerous factors, including.
 Distance between words. If I'm looking for hot near spicy, the string spicy hot has the words at a distance of
0 and the string hot and very spicy has the words at a distance of 2.
 Minimum distance between words. If the search words occur multiple times within the same row, what is the
minimum distance apart?
 Size of columns. If the data columns are 200 words long, a distance of 5 between hot and spicy is considered
nearer than a distance of 2 in a column with only 10 words.

Here's a simple example that uses NEAR

-- This query returns rows that have hot near spicy
SELECT * FROM products
WHERE CONTAINS(*, 'hot near spicy')

The query above returns the same results as the query that looks for hot and spicy. In my small data set, all
occurrences of hot and spicy are near each other.

Each row that meets the conditions of a full-text search is given a ranking value that indicates how "strongly" it meets
the criteria. If our search string includes NEAR, rows that have the search words nearer will be ranked higher. Rows
that contain the search words more often will also be ranked higher. We can also supply our own weighing with an
argument called ISABOUT and assign a weight to certain words. Here's an example:
SELECT *
FROM Products
WHERE CONTAINS(*,'ISABOUT(spicy weight(.8), supreme weight(.2),
hot weight(.1))')

The query asks for rows that contain spicy, supreme, or hot. The word spicy will have four times the weight of the
word supreme, which in turn will have twice the weight of hot. The results don't mean much in terms of the rank. To
see the internal ranking that the Search service has assigned to each qualifying row, we need to use the predicate
CONTAINSTABLE. Here's a similar query with the same search string:
SELECT *
FROM CONTAINSTABLE(Products, *,
'ISABOUT(spicy weight(.8), supreme weight(.1),hot weight(.2))')
This document is created with the unregistered version of CHM2PDF Pilot

The output here is even less meaningful. Here are the first few rows:
KEY RANK
----------- -----------
5 130
3 4
4

The results returned by CONTAINSTABLE report only the internal ranking and the unique index key associated
with each row. However, since we know which column has the unique index, we can join the result of
CONTAINSTABLE with the original table and order by the RANK column. For the products table, the unique key
is ProductID. Here's the query:
SELECT [KEY], RANK, ProductID, ProductName
FROM CONTAINSTABLE(Products, *,'
ISABOUT(spicy weight(.8), supreme weight(.1), hot weight(.2))') C
JOIN Products P
ON P.productID = C.[KEY]
ORDER BY RANK DESC

Here are the complete results:

KEY RANK ProductID ProductName
---- ------ ----------- -----------------------------------------------
65 58 65 Hot and Spicy Louisiana Fiery Hot Pepper Sauce
15 53 15 Hot and Spicy Genen Shouyu Supreme
63 53 63 Spicy Hot Vegie-spread Supreme
20 53 20 Supreme Hot and Spicy Sir Rodney's Marmalade
21 53 21 Spicy Hot Sir Rodney's Scones Supreme
28 53 28 Supreme Spicy Hot Rössle Sauerkraut
75 53 75 Hot and Spicy Rhönbräu Klosterbier Supreme
30 53 30 Hot and Spicy Nord-Ost Matjeshering Supreme
40 53 40 Supreme Hot and Spicy Boston Crab Meat
42 53 42 Spicy Hot Singaporean Hokkien Fried Mee Supreme
45 53 45 Hot and Spicy Rogede sild Supreme
56 53 56 Supreme Spicy Hot Gnocchi di nonna Alice
60 53 60 Hot and Spicy Camembert Pierrot Supreme
55 48 55 Hot and Spicy Pâté chinois
49 48 49 Spicy Hot Maxilaku
50 48 50 Hot and Spicy Valkoinen suklaa
35 48 35 Hot and Spicy Steeleye Stout
77 48 77 Spicy Hot Original Frankfurter grüne Soße
25 48 25 Hot and Spicy NuNuCa Nuß-Nougat-Creme
70 48 70 Hot and Spicy Outback Lager
10 48 10 Hot and Spicy Ikura
7 48 7 Spicy Hot Uncle Bob's Organic Dried Pears
14 48 14 Spicy Hot Tofu
5 48 5 Hot and Spicy Chef Anton's Gumbo Mix
9 43 9 Mishi Kobe Niku Supreme Extra Spicy
18 43 18 Carnarvon Tigers Supreme Extra Spicy
27 43 27 Schoggi Schokolade Supreme Extra Spicy
72 43 72 Mozzarella di Giovanni Supreme Extra Spicy
36 43 36 Inlagd Sill Supreme Extra Spicy
54 43 54 Tourtière Supreme Extra Spicy
66 14 66 Louisiana Hot Spiced Okra Supreme
8 4 8 Supreme Northwoods Cranberry Sauce
3 4 3 Aniseed Syrup Supreme
4 4 4 Supreme Chef Anton's Cajun Seasoning
6 4 6 Grandma's Boysenberry Spread Supreme
16 4 16 Supreme Pavlova
This document is created with the unregistered version of CHM2PDF Pilot

12 4 12 Queso Manchego La Pastora Supreme

32 4 32 Supreme Mascarpone Fabioli
33 4 33 Geitost Supreme
69 4 69 Gudbrandsdalsost Supreme
64 4 64 Supreme Wimmers gute Semmelknödel
24 4 24 Guaraná Fantástica Supreme
68 4 68 Supreme Scottish Longbreads
52 4 52 Supreme Filo Mix
76 4 76 Supreme Lakkalikööri
51 4 51 Manjimup Dried Apples Supreme
39 4 39 Chartreuse verte Supreme
48 4 48 Chocolade Supreme
44 4 44 Supreme Gula Malacca
57 4 57 Ravioli Angelo Supreme

(50 row(s) affected)

Note that the rows containing multiple occurrences of some of the words are returned first, followed by rows
containing all three search words, rows containing only spicy, and then rows containing only supreme.

Rank can be a value between 0 and 1000, but because of the complex algorithm used to determine rank, you'll never
get a rank of 1000. In addition to the complex rules for determining what NEAR means, other rules come into play
during a search for a simple word. If a word occurs in every row in a table, it carries less weight internally than a word
that only occurs a few times. Here's an example:
SELECT [KEY], RANK, ProductID, ProductName
FROM CONTAINSTABLE(Products, *,'hot or tofu ') C
JOIN Products P
ON P.productID = C.[KEY]
ORDER BY RANK DESC

Because hot occurs in many of the rows of the products table, it is given less weight than tofu, which occurs only in
two rows. This is the Search service's own weighting, which is applied even if you don't use the ISABOUT function to
specify a weight. Rows that contain both hot and tofu are ranked highest, followed by rows containing just tofu,
rows that contain just hot, and then rows that contain hot more than once!

FREETEXT and FREETEXTTABLE Predicates

The FREETEXT and FREETEXTTABLE predicates allow you to compare one text value with another to determine
how similar they are. You can enter just a string of individual words, or you can grab a value from another table to use
for comparison. Here's a simple example:
SELECT *
FROM Products
WHERE FREETEXT (*, 'I love hot and spicy scones')

The Search service breaks the FREETEXT argument apart into meaningful words and compares those words to the
words in the full-text index. FREETEXT includes an implied "inflectional" function, but you cannot add an explicit
function to FREETEXT. So this query will find the row with seasoning:
SELECT *
FROM Products
WHERE FREETEXT (*, 'Highly seasoned foods')

Again, by default the rows don't come back in any order. However, the FREETEXTTABLE predicate, which
functions analogously to the CONTAINSTABLE predicate, allows us to obtain the ranks and join back to the base
This document is created with the unregistered version of CHM2PDF Pilot

table:
SELECT [Key], Rank, ProductID, ProductName
FROM FREETEXTTABLE (Products, *,'I love hot and spicy scones') F
JOIN Products P
ON P.productID = F.[KEY]
ORDER BY RANK DESC

Here are the first few rows of the result:

Key Rank ProductID ProductName
----- ----- ----------- ----------------------------------------------
21 56 21 Spicy Hot Sir Rodney's Scones Supreme
65 18 65 Hot and Spicy Louisiana Fiery Hot Pepper Sauce
5 12 5 Hot and Spicy Chef Anton's Gumbo Mix
10 12 10 Hot and Spicy Ikura
7 12 7 Spicy Hot Uncle Bob's Organic Dried Pears
14 12 14 Spicy Hot Tofu

Sometimes hundreds or even thousands of rows will meet your criteria to some degree. SQL Server 2000 lets you
ask the Search service to give you only a few rows, based on the internal ranking. You can use a fourth argument to
the CONTAINSTABLE or FREETEXTTABLE predicates, which is an integer described as TOP n BY RANK.
Here is the CONTAINSTABLE query, with the addition of an argument to ask for only the TOP 10 rows:
SELECT [KEY], RANK, ProductID, ProductName
FROM CONTAINSTABLE(Products, *,'
ISABOUT(spicy weight(.8), supreme weight(.1), hot weight(.2))', 10) C
JOIN Products P
ON P.productID = C.[KEY]
ORDER BY RANK DESC

Be very careful with this TOP n BY RANK argument if you're applying other search conditions to your
CONTAINSTABLE or FREETEXTTABLE queries. The full-text search is done first and the limited rows are
returned to SQL Server, and only then is your own WHERE condition applied. Suppose I want to see the names of
products that contain spicy, supreme, or hot, as weighted in the preceding query, but I'm interested only in products
in category 6. A lookup in the category table tells me that this is meat and poultry. It turns out that none of the top 10
ranking rows is in category 6, so this query will find no rows:
SELECT [KEY], RANK, ProductID, ProductName
FROM CONTAINSTABLE(Products, *,'
ISABOUT(spicy weight(.8), supreme weight(.1), hot weight(.2))', 10) C
JOIN Products P
ON P.productID = c.[KEY]
WHERE categoryID = 6
ORDER BY RANK DESC

If we want our condition to be applied first, and then get the top 10 rows, we have to use SQL Server's own TOP
function:
SELECT TOP 10 [KEY], RANK, ProductID, ProductName
FROM CONTAINSTABLE(Products, *,'
ISABOUT(spicy weight(.8), supreme weight(.1), hot weight(.2))') C
JOIN Products P
ON P.productID = C.[KEY]
WHERE categoryID = 6
ORDER BY RANK DESC
This document is created with the unregistered version of CHM2PDF Pilot

More About Full-Text Indexes

I've already told you about the FULLTEXTCATALOGPROPERTY function, which gives you some important
information about the status of your catalog population. The OBJECTPROPERTY function also has some arguments
that allow you to get full-text index information from individual tables. In some cases, you might find these more useful
than FULLTEXTCATALOGPROPERTY, and I recommend that you use them when possible. Future versions of
full-text indexes might minimize or hide the catalogs, so you'll work only with the individual tables. Table 10-17 lists
the OBJECTPROPERTY arguments that deal with full-text indexes.

Table 10-17. Arguments of OBJECTPROPERTY.

OBJECTPROPERTY Argument Description
The table has full-text back ground update index enabled.
TableFullTextBackgroundUpdateIndexOn 1 = TRUE
0 = FALSE
The ID of the full-text catalog in which the full-text index
data for the table resides.
Nonzero = Full-text catalog ID, associated with the
TableFulltextCatalogId
unique in dex that identifies the rows in a full-text indexed
table.
0 = Table is not full-text indexed.
The table has full-text change tracking enabled.
TableFullTextChangeTrackingOn 1 = TRUE
0 = FALSE
The ID of the column associated with the single-column
unique index that is participating in the full-text index
TableFulltextKeyColumn
definition.
0 = Table is not full-text indexed.
0 = No population
TableFullTextPopulateStatus 1 = Full population
2 = Incremental population
The table has an active full-text index.
TableHasActiveFulltextIndex 1 = True
0 = False

So, to find out if my products table has background index updating enabled, I can execute the following query:
SELECT OBJECTPROPERTY ( object_id('products'),
'TableFullTextBackgroundUpdateIndexOn')

A returned value of 1 means that background index updating is enabled, a value of 0 means no, and a NULL means
that I typed something wrong.

Performance Considerations for Full-Text Indexes

If you are full-text indexing on tables that have less than a million rows, you have little performance tuning to do. If
you create full-text indexes on SQL Server tables that contain millions of rows, you need to be aware that this will
sustain heavy read and write activity. You should consider configuring your SQL Server and the directory where your
catalogs are stored to maximize disk I/O performance by load balancing across multiple disk drives.

Hardware and Operating System Configuration

This document is created with the unregistered version of CHM2PDF Pilot

For your hardware, you should consider multiple fast processors and lots of additional memory. If you'll be doing a
lot of full-text indexing and searching, 1 GB should be the minimum memory. You'll also want multiple disk controllers
with several channels each, if possible, or, at a minimum, multiple channels on a single controller. Consider using a
very fast disk system with striping and no fault tolerance to maximize read and write performance.

If you're installing SQL Server on a Windows NT Server, your page file will need to be 1.5 to 2 times the amount of
available physical RAM. This amount can be reduced on Windows 2000 if you have larger amounts of RAM. The
page file should be placed on its own drive with its own controller.

SQL Server Configuration

After a full population of a table with more than a million rows, the fastest, most efficient method of maintaining your
indexes is to use the change-tracking feature with the background update option. If you must do a full or incremental
population or update the changes in a single operation rather than in the background, the population should be done
during (or scheduled for) periods of low system activity, typically during database maintenance windows, if you have
them.

When you determine whether to use one or multiple full-text catalogs, you must take into account both maintenance
and searching considerations. There is a trade-off between performance and maintenance when considering this
design question with large SQL tables, and I suggest that you test both options for your environment. If you choose to
have multiple tables in one catalog, you'll incur some overhead because of longer running full-text search queries, but
the main penalty will be that any population will take longer because all tables in the catalog will need to be updated. If
you choose to have a single SQL table per catalog, you'll have the overhead of maintaining separate catalogs. You
also should be aware that there is a limit of 256 full-text catalogs per machine. The Search service imposes this limit,
and there is only one Search service per machine. The limit is not related to the number of SQL Server instances you
might have.
NOTE
You should also be aware that because the Search service is a component external to SQL Server, it can change
when you upgrade other components. For example, Windows NT 4 and Windows 2000 have slightly different
Search service implementations, with different word-breaking algorithms. You might occasionally see slightly different
results when you run on these different platforms. You'll need to check the documentation to see if products you install
on your server after installing SQL Server 2000 use the Microsoft Search service.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
This chapter discussed the programming extensions beyond typical implementations of SQL that make
Transact-SQL a specialized programming language. These extensions include control of flow, conditional branching,
looping, variables, logical operations, bitwise operations, and a variety of scalar functions. The extensions let you write
sophisticated routines directly in Transact-SQL and execute them directly in the SQL Server engine. Without them,
you would have to write much more logic into client applications, which would hurt performance because more
conversation would need to take place across the network. We also took a long look at how to use SQL Server
2000 in conjunction with the Search service to maintain and query full-text indexes on your data.

From here, we'll move on to using the Transact-SQL extensions in stored procedures and triggers.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 11

Batches, Stored Procedures, and

Functions
In this chapter, we'll use Transact-SQL for more than interactive queries. When you send a query to the server,
you're sending a command batch to SQL Server. But we can do even more. For example, we can wrap up
commands in a module that can be stored and cached at the server for later reuse. Modules that contain data
modification operations can be created as stored procedures; modules that return a value or a result set without
affecting other data in your database can be created as user-defined functions.

I'll describe the creation, use, and internal storage of stored procedures and user-defined functions. The ANSI SQL
Standard refers to both stored procedures and user-defined functions as "routines"; I'll use that term when I need to
refer to both these types of objects together. SQL Server allows us to create another type of stored module that is
automatically executed when a request is made to modify data; this type of object is called a trigger. However, the
SQL Standard does not consider triggers in its definition of "routines," so I won't either. I'll cover the details of trigger
creation and use in Chapter 12.
This document is created with the unregistered version of CHM2PDF Pilot

34

Batches
A batch is one or more SQL Server commands that are dispatched and executed together. Because every batch sent
from the client to the server requires handshaking between the two, sending a batch instead of sending separate
commands can be more efficient. Even a batch that doesn't return a result set (for example, a single INSERT
statement) requires at least an acknowledgment that the command was processed and a status code for its level of
success. At the server, the batch must be received, queued for execution, and so on.

Although commands are grouped and dispatched together for execution, each command is distinct from the others.
Let's look at an example. Suppose you need to execute 150 INSERT statements. Executing all 150 statements in one
batch requires the processing overhead to be incurred once rather than 150 times. In one real-world situation, an
application that took 5 to 6 seconds to complete 150 individual INSERT statements was changed so that all 150
statements were sent as one batch. The processing time decreased to well under 0.5 second, more than a tenfold
improvement. And this was on a LAN, not on a slow network like a WAN or the Internet, where the improvement
would have been even more pronounced. Try running an application with 150 batches to insert 150 rows over the
Internet, and you'll be really sorry!

Using a batch is a huge win. By using the Transact-SQL constructs presented in Chapter 10, such as conditional
logic and looping, you can often perform sophisticated operations within a single batch and eliminate the need for an
extensive conversation between the client and the server. Those operations can also be saved on the server as a
stored procedure, which allows them to execute even more efficiently. If your operations are returning values rather
than modifying them, you can also save the code as a user-defined function. Using batches and stored modules to
minimize client/server conversations is crucial for achieving high performance. And now that more applications are
being deployed on slower networks—such as WANs, the Internet, and dial-up systems—instead of on LANs only,
using batches and stored modules is crucial.

Every SELECT statement (except those used for assigning a value to a variable) generates a result set. Even a
SELECT statement that finds zero rows returns a result set that describes the columns selected. Every time the server
sends a result set back to the client application, it must send metadata as well as the actual data. The metadata
describes the result set to the client. You can think of metadata in this way: "Here's a result set with eight columns. The
first column is named last_name and is of type char(30). The second column is…"

Obviously, then, executing a single SELECT statement with a WHERE clause formulated to find, in one fell swoop,
all 247 rows that meet your criteria is much more efficient than separately executing 247 SELECT statements that
each return one row of data. In the former case, one result set is returned. In the latter case, 247 result sets are
returned. The performance difference is striking.

Although batches seem like an obvious necessity, many programmers still write applications that perform poorly
because they don't use batches. This problem is especially common among developers who have worked on ISAM
or similar sequential files that do row-at-a-time processing. Unlike ISAM, SQL Server works best with sets of data,
not individual rows of data, so that you can minimize conversations between the server and the client application.

There's actually a middle ground between using one SELECT statement that returns all your rows and using an
individual batch to return each row. Batches can contain multiple statements, each one returning its own metadata, so
the ideal solution is to minimize your total number of statements. Multiple SELECTs within a single batch are still
better than having each statement in its own batch because each batch has the overhead of handshaking between the
client and server.

Here is a simple batch, issued from SQL Query Analyzer. Even though three unrelated operations are being
performed, we can package them in a single batch to conserve bandwidth:
This document is created with the unregistered version of CHM2PDF Pilot

INSERT authors VALUES (etc.)

SELECT * FROM authors
UPDATE publishers SET pub_id= (etc.)
<click the Execute Query button>

All the statements within a single batch are parsed as a unit. This means that if you have a syntax error in any of the
statements, none of the statements will execute. For example, consider this batch of two statements, which you can try
to execute from SQL Query Analyzer:
SELECT * FROM sales
SELECT * FOM titleauthor
<click the Execute Query button>

When you execute this batch, you get the following error:
Server: Msg 170, Level 15, State 1, Line 2
Line 2: Incorrect syntax near 'FOM'.

Even though the first SELECT statement is perfectly legal, no data is returned because of the error in the second
statement. If we separated this into two different batches, the first one would return a result set and the second one
would return an error.

Now suppose that instead of mistyping the keyword FROM as FOM, we had mistyped the name of the table:
SELECT * FROM sales
SELECT * FROM titleautor
<click the Execute Query button>

SQL Server 2000 provides delayed name resolution, in which object names aren't resolved until execution time. In
this example, SQL Server returns the data from the first SELECT, followed by this error message:
Server: Msg 208, Level 16, State 1, Line 1
Invalid object name 'titleautor'.

However, if we had mistyped the object name in the first statement, execution of the batch would stop when the error
was encountered. No data would be returned by any subsequent valid queries.

Note that delayed object resolution does not apply to column names. If you changed the second query to one that
used a valid table name but specified an invalid column in that table, the error would be detected at compile time and
none of the statements in the batch would be executed. Here's an example:
SELECT * FROM sales
SELECT lastname FROM titleauthor
<click the Execute Query button>

Normally, everything in SQL Query Analyzer's query window is considered a single batch and is sent to SQL Server
for parsing, compiling, and execution when you click the green EXECUTE button. However, SQL Query Analyzer
provides two ways around this. First, you can highlight a section of code in the query window so that when you click
the EXECUTE button, only the highlighted text is sent as a batch to SQL Server. Alternatively, you can include the
word GO between your queries.

If you use either of the text-based query tools (OSQL or ISQL), there's no green Execute Query button to click.
This document is created with the unregistered version of CHM2PDF Pilot

When you use these tools, you must type GO on a line by itself to indicate that everything you've typed up to that
point is a batch. The tool will then send that batch to SQL Server for processing.

GO isn't an SQL command or keyword. It's the end-of-batch signal understood only by certain client tools. The
client interprets it to mean that everything since the last GO should be sent to the server for execution. SQL Server
never sees the GO command and has no idea what it means. With a custom application, a batch is executed with a
single SQLExecute from ODBC (or dbsqlexec from DB-Library).

If you include the GO command in the query window, SQL Query Analyzer breaks up your statement into the
indicated batches behind the scenes. Each batch (as marked by the GO command) is sent individually to SQL Server.

A collection of batches that are frequently executed together is sometimes called a script. Most of the client tools
provide a mechanism for loading a script that you've saved to a text file and for executing it. In SQL Query Analyzer,
you can use the File/Open command to load a script. From the command-line OSQL or ISQL programs, we can
specify the /i flag followed by a filename to indicate that the SQL Server batches to execute should come from the
specified file. Alternatively, if we're using OSQL or ISQL interactively, we can read in a file containing one or more
SQL statement by typing :r followed by the filename. (See SQL Server Books Online for details about using OSQL
and ISQL.)

The fact that the client tool and not SQL Server processes GO can lead to some unexpected behavior. Suppose you
have a script containing several batches. During testing, you want to comment out a couple of the batches to ignore
them for the time being. Your commented script might look something like this:
SELECT * FROM authors
/*
GO
SELECT * FROM sales
GO
SELECT * FROM publishers
GO
*/
SELECT * FROM titles
GO

The intention here was to comment out the SELECT from the sales and publishers tables and to run the SELECT
from authors and titles as a single batch. However, if you run this script from SQL Query Analyzer, you'll get exactly
the opposite behavior! That is, you'll see the data from the sales and publishers tables but not from authors or titles.
If you look at this script from the perspective of the client tool, the behavior makes sense. The tool doesn't try to
interpret any of your SQL statements; it just breaks the statements into batches to be sent to SQL Server. A batch is
marked by a GO command at the beginning of a line.

So the above script contains four batches. The first batch (everything before the first GO) is:
SELECT * FROM authors
/*

SQL Server generates an error message because there's an open comment with no corresponding close comment.

The second and third batches are:

SELECT * FROM sales

and
SELECT * FROM publishers
This document is created with the unregistered version of CHM2PDF Pilot

Both of these batches are perfectly legal, and SQL Server can process them and return results.

The fourth batch is:

*/
SELECT * FROM titles

SQL Server also generates an error for this last one because it has a close comment without an open comment
marker, and no data is returned.

If you want to comment out statements within a script that can contain the end-of-batch GO command, you should
use the alternative comment marker—the double dash—in front of every GO. Alternatively, you can just use the
double dash in front of every line you want to comment out. Your script would then look like this:
SELECT * FROM authors
-- GO
-- SELECT * FROM sales
-- GO
-- SELECT * FROM publishers
-- GO
SELECT * FROM titles
GO

SQL Query Analyzer makes it easy to comment out a group of lines as in the code above. You can highlight all the
lines to be commented out, and from the Edit menu, choose Advanced/Comment Out. The keystroke combination to
perform the same action is Ctrl-Shift-C. With this revised script, the client tool won't recognize the GO as the
end-of-batch marker because it's not the first thing on a line. The client will consider this script to be one single batch
and send it to SQL Server as such.
This document is created with the unregistered version of CHM2PDF Pilot

34

Routines
In SQL Server 2000, stored procedures and user-defined functions are considered routines. The ANSI SCHEMA
VIEW called ROUTINES contains information about all your procedures and user-defined functions, including the
text that makes up the routine's definition. You can inspect this view and get information about all the routines in a
database by selecting from this ANSI SCHEMA VIEW. I ran the following in the master database:
SELECT routine_type, routine_name FROM INFORMATION_SCHEMA.ROUTINES

Here are the partial results:

routine_type routine_name
-------------------- -------------------------------
PROCEDURE sp_MSgetrowmetadata
PROCEDURE sp_MSindexspace
FUNCTION fn_sqlvarbasetostr
FUNCTION fn_helpcollations
PROCEDURE sp_describe_cursor_tables
PROCEDURE sp_MSIfExistsRemoteLogin
FUNCTION fn_trace_getinfo
PROCEDURE sp_MSmakedynsnapshotvws
FUNCTION fn_chariswhitespace
FUNCTION fn_generateparameterpattern
FUNCTION fn_skipparameterargument
PROCEDURE sp_MSmakeexpandproc
FUNCTION fn_removeparameterwithargument

As you'll see in more detail later, stored procedures and functions are invoked in different ways. Simply put, stored
procedures can be invoked anywhere a statement can be executed, while a function is invoked inside another
statement. You can use scalar functions anywhere that your SQL command expects a single value, and you can use
table-valued functions in the FROM clause to indicate the rowset to be processed.
This document is created with the unregistered version of CHM2PDF Pilot

34

Stored Procedures
Understanding batches is a crucial prerequisite to understanding stored procedures, but you shouldn't confuse the
two. Batches involve communication between a client program and SQL Server, and stored procedures are objects
that exist only on the server. A stored procedure always executes within context of a single batch, but a single batch
can contain calls to multiple stored procedures as well as other commands that are not part of stored procedures.

We've actually seen stored procedures in use quite a few times already. For example, we've looked at the output to
the system stored procedures sp_help, sp_helpconstraint and sp_helpindex. The stored procedures that you can
create are not too different from the system-supplied procedures. After I tell you a bit more about stored procedures,
you'll be able to see exactly what the similarities and differences are between the procedures you write and the ones
Microsoft supplies with SQL Server.

To create a stored procedure, you take a batch and wrap it inside a CREATE PROCEDURE statement. The
procedure's definition can consist of any Transact-SQL commands, with nothing special except for declaring which
parameters are passed to the procedure.

To demonstrate how easy it is to create a stored procedure, I've written a simple procedure called get_author that
takes one parameter, the author ID, and returns the names of any authors that have IDs equal to whatever character
string is passed to the procedure:
CREATE PROC get_author @au_id varchar(11)
AS
SELECT au_lname, au_fname
FROM authors
WHERE au_id=@au_id

This procedure can be subsequently executed with syntax like the following:
EXEC get_author '172-32-1176'

or
EXEC get_author @au_id='172-32-1176'

As you can see, you can pass the parameters anonymously by including specific values for them. If the procedure is
expecting more than one parameter and you want to pass them anonymously, you must specify their values in the
order that the parameters were listed in the CREATE PROCEDURE statement. Alternatively, you can explicitly name
the parameters so that the order in which they're passed isn't important.

If executing the procedure is the first statement in the batch, using the keyword EXEC is optional. However, it's
probably best to use EXEC (or EXECUTE) so that you won't wind up wondering why your procedure wouldn't
execute (only to realize later that it's no longer the first statement of the batch).

In practice, you'll probably want a procedure like the one above to be a bit more sophisticated. Perhaps you'll want
to search for author names that begin with a partial ID that you pass. If you choose not to pass anything, the
procedure should show all authors rather than return an error message stating that the parameter is missing.

Or maybe you'd like to return some value as a variable (distinct from the result set returned and from the return code)
This document is created with the unregistered version of CHM2PDF Pilot

that you can use to check for successful execution. You can do this by passing an output parameter, which has a
pass-by-reference capability. Passing an output parameter to a stored procedure is similar to passing a pointer when
you call a function in C. Rather than passing a value, you pass the address of a storage area in which the procedure
will cache a value. That value is subsequently available to the SQL batch after the stored procedure has executed.

For example, you might want an output parameter to tell you the number of rows returned by a SELECT statement.
While the @@ROWCOUNT system function provides this information, it's available for only the last statement
executed. If the stored procedure has executed many statements, you must put this value away for safekeeping. An
output parameter provides an easy way to do this.

Here's a simple procedure that selects all the rows from the authors table and all the rows from the titles table. It
also sets an output parameter for each table based on @@rowcount. The calling batch retrieves the
@@ROWCOUNT values through the variables passed as the output parameters.
CREATE PROC count_tables @authorcount int OUTPUT,
@titlecount int OUTPUT
AS
SELECT * FROM authors
SET @authorcount=@@ROWCOUNT
SELECT * FROM titles
SET @titlecount=@@ROWCOUNT
RETURN(0)

The procedure would then be executed like this:

DECLARE @a_count int, @t_count int
EXEC count_tables @a_count OUTPUT, @t_count OUTPUT

TIP
Parameters passed to a routine are treated much like local variables. Variables declared in a stored procedure are
always local, so we could have used the same names for both the variables in the procedure and those passed by the
batch as output parameters. In fact, this is probably the most common way to invoke them. However, the variables
passed into a routine don't need to have the same name as the associated parameters. Even with the same name, they
are in fact different variables because their scoping is different.

This procedure returns all the rows from both tables. In addition, the variables @a_count and @t_count retain the
row counts from the authors and titles tables, respectively. After the EXEC statement inside the batch, we can look
at the values in the two variables passed to procedure to see what values they ended up with:
SELECT authorcount=@a_count, titlecount=@t_count

Here's the output for this SELECT:

authorcount titlecount
----------- ----------
23 18

When you create a stored procedure, you can reference a table, a view, or another stored procedure that doesn't
currently exist. In the latter case, you'll get a warning message informing you that a referenced object doesn't exist. As
long as the object exists at the time the procedure is executed, all will be fine.

Nested Stored Procedures

Stored procedures can be nested and can call other procedures. A procedure invoked from another procedure can
This document is created with the unregistered version of CHM2PDF Pilot

also invoke yet another procedure. In such a transaction, the top-level procedure has a nesting level of 1. The first
subordinate procedure has a nesting level of 2. If that subordinate procedure subsequently invokes another stored
procedure, the nesting level is 3, and so on, to a limit of 32 nesting levels. If the 32-level limit is reached, a fatal error
occurs, the batch is aborted, and any open transaction is rolled back.

The nesting-level limit prevents stack overflows that can result from procedures recursively calling themselves
infinitely. The limit allows a procedure to recursively call itself only 31 subsequent times (for a total of 32 procedure
calls). To determine how deeply a procedure is nested at runtime, you can select the value of the system function
@@NESTLEVEL.

Unlike with nesting levels, SQL Server has no practical limit on the number of stored procedures that can be invoked
from a given stored procedure. For example, a main stored procedure can invoke hundreds of subordinate stored
procedures. If the subordinate procedures don't invoke other subordinate procedures, the nesting level never reaches
a depth greater than 2.

An error in a nested (subordinate) stored procedure isn't necessarily fatal to the calling stored procedure. When you
invoke a stored procedure from another stored procedure, it's smart to use a RETURN statement and check the
return value in the calling procedure. In this way, you can work conditionally with error situations (as shown in the
upcoming factorial example).

Recursion in Stored Procedures

Stored procedures can perform nested calls to themselves, also known as recursion. Recursion is a technique by
which the solution to a problem can be expressed by applying the solution to subsets of the problem. Programming
instructors usually demonstrate recursion by having students write a factorial program using recursion to display a
table of factorial values for 0! through 10!. Recall that a factorial of a positive integer n, written as n!, is the product of
all integers from 1 through n. For example:
8! = 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1 = 40320

(Zero is a special case—0! is defined as equal to 1.)

We can write a stored procedure that computes factorials, and we can do the recursive programming assignment in
Transact-SQL:
-- Use Transact-SQL to recursively calculate factorial
-- of numbers between 0 and 12.
-- Parameters greater than 12 are disallowed because the
-- result overflows the bounds of an int.

CREATE PROC factorial @param1 int

AS
DECLARE @one_less int, @answer int
IF (@param1 < 0 OR @param1 > 12)
BEGIN
-- Illegal parameter value. Must be between 0 and 12.
RETURN -1
END

IF (@param1=0 or @param1=1)
SELECT @answer=1
ELSE
BEGIN
SET @one_less=@param1 - 1
EXEC @answer=factorial @one_less -- Recursively call itself
IF (@answer= -1)
BEGIN
This document is created with the unregistered version of CHM2PDF Pilot

RETURN -1
END

SET @answer=@answer * @param1

IF (@@ERROR <> 0)
RETURN -1
END

RETURN(@answer)

Note that when the procedure is initially created, a warning message like the one shown here will indicate that the
procedure references a procedure that doesn't currently exist (which is itself in this case):
Cannot add rows to sysdepends for the current stored procedure
because it depends on the missing object 'factorial'. The stored
procedure will still be created.

Once the procedure exists, we can use it to display a standard factorial:

DECLARE @answer int, @param int
SET @param=0
WHILE (@param <= 12) BEGIN
EXEC @answer = factorial @param
IF (@answer= -1) BEGIN
RAISERROR('Error executing factorial procedure.', 16, -1)
RETURN
END
PRINT CONVERT(varchar, @param) + '! = ' + CONVERT(varchar(50), @answer)
SET @param=@param + 1
END

Here's the output table:

0! = 1
1! = 1
2! = 2
3! = 6
4! = 24
5! = 120
6! = 720
7! = 5040
8! = 40320
9! = 362880
10! = 3628800
11! = 39916800
12! = 479001600

We stopped at 12! in the factorial procedure because 13! is 6,227,020,800, which exceeds the range of a 32-bit
(4-byte) integer. Even if we changed the parameter @answer to be type decimal, we would still be limited to 12!
because the value included in a RETURN statement must be type int.

Here's another version of the procedure that uses an output parameter for the answer and introduces a new variable
to hold the returned status of the procedure. You can use a decimal datatype with a scale of 0 as an alternative to int
for integer operations that require values larger than the 4-byte int can handle.
CREATE PROC factorial @param1 decimal(38,0), @answer decimal(38,0) output
AS
DECLARE @one_less decimal(38,0), @status int
This document is created with the unregistered version of CHM2PDF Pilot

IF (@param1 < 0 OR @param1 > 32)

BEGIN
-- Illegal parameter value. Must be between 0 and 32.
RETURN -1
END

IF (@param1=0 or @param1=1)
SET @answer=1
ELSE
BEGIN
SET @one_less=@param1 - 1
EXEC @status=factorial @one_less, @answer output
-- Recursively call itself
IF (@status= -1)
BEGIN
RETURN -1
END

SET @answer=@answer * @param1

IF (@@ERROR <> 0)
RETURN -1
END

RETURN 0

To call this procedure, we can use the following code:

DECLARE @answer decimal(38,0), @param int
SET @param=0
WHILE (@param <= 32) BEGIN
EXEC factorial @param, @answer output
IF (@answer= -1)
BEGIN
RAISERROR('Error executing factorial procedure.', 16, -1)
RETURN
END
PRINT CONVERT(varchar, @param) + '! = ' + CONVERT(varchar(50), @answer)
SET @param=@param + 1
END

Even though this procedure has removed the range limit on the result by using a decimal datatype to hold the answer,
we can only go to 32! because we'd reach the maximum nesting depth of 32, which includes recursive calls. If we
took out the condition OR @param1 > 32, this would allow an initial parameter > 32, which would result in a nesting
level that exceeded the maximum. We would then get this error:
Server: Msg 217, Level 16, State 1, Procedure factorial, Line 17
Maximum stored procedure, function, trigger, or view-nesting level
exceeded (limit 32).

In C, you need to be sure that you don't overflow your stack when you use recursion. Using Transact-SQL shields
you from that concern, but it does so by steadfastly refusing to nest calls more than 32 levels deep. You can also
watch @@NESTLEVEL and take appropriate action before reaching the hard limit. As is often the case with a
recursion problem, you can perform an iterative solution without the restriction of nesting or worries about the stack.

Here's an iterative approach. To illustrate that there's no restriction of 32 levels because it's simple iteration, we'll go
one step further, to 33!. We'll stop at 33! because 34! would overflow the precision of a numeric(38,0) variable. So
even though the iterative approach removes the nesting level limit, we can only go one step further because of the
This document is created with the unregistered version of CHM2PDF Pilot

range limitations of SQL Server's exact numeric datatypes:

-- Alternative iterative solution does not have the restriction
-- of 32 nesting levels
CREATE PROC factorial2 @param1 int, @answer NUMERIC(38,0) OUTPUT
AS
DECLARE @counter int
IF (@param1 < 0 OR @param1 > 33)
BEGIN
RAISERROR ('Illegal Parameter Value. Must be between 0 and 33',
16, -1)
RETURN -1
END

SET @counter=1 SET @answer=1

WHILE (@counter < @param1 AND @param1 <> 0 )

BEGIN
SET @answer=@answer * (@counter + 1)
SET @counter=@counter + 1
END

RETURN
GO

DECLARE @answer numeric(38, 0), @param int

SET @param=0
WHILE (@param <= 32)
BEGIN
EXEC factorial2 @param, @answer OUTPUT
PRINT CONVERT(varchar(50), @param) + '! = '
+ CONVERT(varchar(50), @answer)
SET @param=@param + 1
END

And here's the output :

0! = 1
1! = 1
2! = 2
3! = 6
4! = 24
5! = 120
6! = 720
7! = 5040
8! = 40320
9! = 362880
10! = 3628800
11! = 39916800
12! = 479001600
13! = 6227020800
14! = 87178291200
15! = 1307674368000
16! = 20922789888000
17! = 355687428096000
18! = 6402373705728000
19! = 121645100408832000
20! = 2432902008176640000
21! = 51090942171709440000
22! = 1124000727777607680000
23! = 25852016738884976640000
24! = 620448401733239439360000
25! = 15511210043330985984000000
This document is created with the unregistered version of CHM2PDF Pilot

26! = 403291461126605635584000000
27! = 10888869450418352160768000000
28! = 304888344611713860501504000000
29! = 8841761993739701954543616000000
30! = 265252859812191058636308480000000
31! = 8222838654177922817725562880000000
32! = 263130836933693530167218012160000000
33! = 8683317618811886495518194401280000000

Because this factorial procedure is really producing a single valued result, we could have created it as a user-defined
function. I'll show you how to do that a little later in this chapter.

Stored Procedure Parameters

Stored procedures take parameters, and you can give parameters default values. If you don't supply a default value
when you create the procedure, an actual parameter will be required when the procedure is executed. If you don't
pass a required parameter, you'll get an error like the following, which occurs when I try to call the factorial procedure
without a parameter:
Server: Msg 201, Level 16, State 3, Procedure factorial, Line 0
Procedure 'factorial' expects parameter '@param1', which was not supplied.

You can pass values by explicitly naming the parameters or by furnishing all the parameter values anonymously but in
correct positional order. You can also use the keyword DEFAULT as a placeholder when you pass parameters. You
can also pass NULL as a parameter (or define it as the default). Here's a simple example:
CREATE PROCEDURE pass_params
@param0 int=NULL, -- Defaults to NULL
@param1 int=1, -- Defaults to 1
@param2 int=2 -- Defaults to 2
AS
SELECT @param0, @param1, @param2
GO

EXEC pass_params -- PASS NOTHING - ALL Defaults

(null) 1 2

EXEC pass_params 0, 10, 20 -- PASS ALL, IN ORDER

0 10 20

EXEC pass_params @param2=200, @param1=NULL

-- Explicitly identify last two params (out of order)
(null) (null) 200

EXEC pass_params 0, DEFAULT, 20

-- Let param1 default. Others by position.
0 1 20

Note that if you pass parameters by value, you can't leave any gaps. That is, you can't pass only the first and third,
using syntax like this:
EXEC pass_params 0,, 20
-- You'll get a syntax error here

Also, be aware of the difference between leaving a parameter value unspecified, using the keyword DEFAULT, and
passing in an explicit NULL. In the first two cases, the procedure must have a default defined for the parameter in
This document is created with the unregistered version of CHM2PDF Pilot

order to avoid an error. If you pass an explicit NULL, you will not get an error, but if there is a default value for the
parameter, it will not be used. NULL will be the "value" used by the procedure.

Wildcards in Parameters
If you have a parameter that is of type character, you might want to pass an incomplete string as a parameter and
use it for pattern matching. Consider this procedure that accepts an input parameter of type varchar and returns the
title of any book from the titles table in the pubs database for which the title_id matches the input string:
CREATE PROC gettitles
@tid varchar(6) = '%'
-- the parameter default means that if no parameter
-- is specified, the procedure will return all the
-- titles
AS

SELECT title
FROM titles
WHERE title_id LIKE @tid

IF @@rowcount = 0
PRINT 'There are no titles matching your input'

RETURN

We can call this procedure in a number of different ways. You should be aware that in many cases, SQL Server is
very smart about datatype conversion, and the following is an acceptable statement, even without quotes around the
character parameter:
EXEC gettitles bu1032

The parser breaks the command into individual units of meaning, called tokens, by using spaces and other
nonalphanumeric characters as separators. Since the parameter here is completely alphanumeric, the parser
recognizes it as a single token and can then assume that it is a parameter of the procedure. The procedure expects a
parameter of type varchar, so it just interprets the value it receives as a varchar.

If your parameter includes any nonalphanumeric characters, such as dashes or spaces, the parser will not be able to
deal with it appropriately and you'll need to help it by enclosing such parameters in quotes or square brackets. So
note what happens if we call the gettitles procedure with a % in the parameter:
EXEC gettitles bu%

RESULT:
Server: Msg 170, Level 15, State 1, Line 1
Line 1: Incorrect syntax near '%'.

This error is not generated by the stored procedure but by the parser before the procedure is ever called. The parser
is unable to determine that the bu% needs to be treated as a single token. If there are quotes around the parameter,
the parser is satisfied and the procedure can be executed. Because the query inside the procedure uses LIKE instead
of =, the pattern matching can be done and we get back all the titles for which the title_id starts with bu:
EXEC gettitles 'bu%'

RESULTS:
title
---------------------------------------------------------
The Busy Executive's Database Guide
This document is created with the unregistered version of CHM2PDF Pilot

Cooking with Computers: Surreptitious Balance Sheets

You Can Combat Computer Stress!
Straight Talk About Computers

(4 row(s) affected)

If you want your procedure to handle wildcards in parameters, you need to make sure not only that the search
condition uses LIKE, but also that the type of parameter is varchar instead of char. You might not think that this
would be necessary here because all of the title_id values are exactly six characters long. But suppose that in this
gettitles example we had declared @tid to be char(6). This would mean that the procedure would need a parameter
of exactly six characters. If we had passed in the three characters bu%, SQL Server would have had to pad the
parameter with spaces ('bu% ') to get the full six characters. Then, when the comparison was made in the WHERE
clause of the procedure's SELECT statement, SQL Server would look for title_id values that start with bu, are
followed by anything (because of the %), and end with three spaces! Because none of the title_id values in the titles
table end with three spaces, no rows would be returned.

The default of % in the procedure means that if no parameter is specified, the procedure will compare the title_id
with % and return all the rows in the table. This is a useful technique to make the procedure more versatile. Many of
our system procedures work this way; for example, if you execute sp_helpdb with a parameter, that parameter is
considered the name of a database and you'll get back details about that database. With no parameter, sp_helpdb
will return information about all your databases.

However, this technique works only for character string parameters because that is the only datatype that allows
wildcards. What if I wanted to write a procedure to find all books with a particular value for price? If no price is
specified, I want the procedure to return the names of all the books.

One solution is to define a parameter with a default of NULL. I could write the procedure as two completely
separate SELECT statements, one for when the parameter has a non-null value and one without a WHERE clause for
when the parameter is NULL. However, there is another way to do this in a single SELECT statement:
CREATE PROC gettitle_by_price
@cost money = NULL
AS
SELECT price, title
FROM titles
WHERE price = ISNULL(@cost, price)
RETURN

This procedure assigns a default value of NULL to the parameter, which prevents an error from being generated if no
parameter is actually passed in. However, only a single SELECT statement is needed. The ISNULL function will end
up comparing a price to itself if the parameter is NULL, and all values for price will be equal to themselves.

There is one little gotcha here, which is that the result from executing this procedure with no parameter is not exactly
the same as executing a SELECT without a WHERE clause. A SELECT with no WHERE clause will return ALL the
rows. The SELECT with the ISNULL in the WHERE clause will return all rows in which the value of price is equal to
the parameter or equal to the same value for price. But what if price has no value? In the titles table, two rows have
a nonvalue, in other words NULL, for the price, and will not be returned from the above procedure. In this situation,
even setting ANSI_NULLS to OFF will not help. We'll try to find the rows WHERE price = price, and because
NULL is never considered equal to NULL, the rows with NULL for price will not be returned. One alternative would
be to write the WHERE clause like this:
WHERE ISNULL(price, 0) = ISNULL(@cost, ISNULL(price, 0))

If we write the procedure with this condition, the results we will get from executing the procedure with no parameter
This document is created with the unregistered version of CHM2PDF Pilot

will be the same as executing a SELECT without a WHERE clause; that is, all the rows will be returned. However,
the drawback to this solution is that SQL Server might not be able to optimize this query and find a good execution
plan. As we'll see in Chapter 16, if a column is embedded in a function call or other expression, the optimizer is very
unlikely to consider using any indexes on that column. That means if our table has an index on the price column and
we actually do pass in a value for price, SQL Server probably won't choose to use that index to find the rows in the
table with the requested price.
This document is created with the unregistered version of CHM2PDF Pilot

34

User-Defined Functions
Just as I've showed stored procedure examples (using sp_help and so on) in earlier chapters, I've also used
functions. You've seen parameterized functions such as CONVERT, RTRIM, and ISNULL as well as parameterless
functions such as @@SPID. SQL Server 2000 lets you create functions of your own. However, whereas
user-defined procedures are managed similarly to system stored procedures, the functions you create are managed
very differently from the supplied functions. As I've mentioned, the ANSI SCHEMA VIEW called ROUTINES
allows you to see all the procedures and functions in a database. This view shows system-defined stored procedures
but not built-in functions such as the ones you've seen in earlier chapters. You should think of the built-in functions as
almost like built-in commands in the SQL language; they are not listed in any system table, and there is no way for you
to see how they are defined.

Table Variables
To make full use of user-defined functions, it's useful to understand a special type of variable that SQL Server 2000
provides. You can declare a local variable as a table or use a table value as the result set of a user-defined function.
You can think of table as a special kind of datatype. Note that you cannot use table variables as stored procedure or
function parameters, nor can a column in a table be of type table.

Like the other kinds of local variables I told you about in Chapter 10, table variables have a well-defined scope,
which is limited to the procedure, function, or batch in which they are declared. Here's a simple example:
DECLARE @pricelist TABLE(tid varchar(6), price money)
INSERT @pricelist SELECT title_id, price FROM titles
SELECT * FROM @pricelist

The definition of a table variable looks almost like the definition of a normal table, except that you use the word
DECLARE instead of CREATE, and the name of the table variable comes before the word TABLE.

The definition of a table variable can include:

 A column list defining the datatypes for each column and specifying the NULL or NOT NULL property
 PRIMARY KEY, UNIQUE, CHECK, or DEFAULT constraints

The definition of a table variable cannot include:

 Foreign key references to other tables
 Columns referenced by a FOREIGN KEY constraint in another table

Within their scope, table variables can be treated like any other table. All data manipulation statements (SELECT,
INSERT, UPDATE, and DELETE) can be performed on the data in a table variable, with two exceptions:
 You cannot use SELECT INTO to add data to a table variable. This is because SELECT INTO creates a
table, and table variable must be created using DECLARE. For example, you cannot do the following:
SELECT select_list INTO table_variable statements

 You cannot directly INSERT the result set of a stored procedure into a table variable. For example, you
cannot do the following:
INSERT INTO table_variable EXEC stored_procedure
This document is created with the unregistered version of CHM2PDF Pilot

In addition, you need to be aware of the following facts:

 A table variable is not part of a transaction. A ROLLBACK TRANSACTION command will not affect data
added to a table variable. (The same is true of any other nontable variables.)
 You cannot use the CREATE INDEX command to build an index on a table variable.

Scalar-Valued Functions
You can use a scalar-valued function anywhere that your Transact-SQL command is expecting a value. It can take
up to 1024 input parameters but no output parameters. The function can return a value of any datatype except
rowversion (or timestamp), cursor, or table.

Unlike a stored procedure, a function must include the RETURN statement. (In stored procedures, the RETURN is
optional.) You can declare a scalar local variable to hold the return value of the function and then use this variable with
the RETURN statement, or the RETURN statement can include the computation of the return value. The following
two function definitions are equivalent; both will return the average price for books in the titles table for which the type
is the same as the type specified as the input parameter:
CREATE FUNCTION AveragePrice(@booktype varchar(12))
RETURNS money
AS
BEGIN
DECLARE @avg money
SELECT @avg = avg(price)
FROM titles
WHERE type = @booktype

RETURN @avg
END

CREATE FUNCTION AveragePrice2(@booktype varchar(12))

RETURNS money
AS
BEGIN

RETURN ( SELECT avg(price)

FROM titles
WHERE type = @booktype)
END

Invoking a Scalar Function

To invoke a user-defined scalar function, you must specify the owner name. The following query will return an error:
SELECT title_id, price
FROM titles
WHERE price > AveragePrice('business')
AND type = 'business'

RESULT:
Server: Msg 195, Level 15, State 10, Line 3
'AveragePrice' is not a recognized function name.

Here's the correct invocation:

SELECT title_id, price
FROM titles
WHERE price > dbo.AveragePrice('business')
This document is created with the unregistered version of CHM2PDF Pilot

AND type = 'business'

RESULT:
title_id price
-------- ---------------------
BU1032 19.9900
BU7832 19.9900

You can invoke user-defined scalar functions by simply SELECTing their value:
SELECT dbo.AveragePrice ('business')

You can also use the keyword EXECUTE to invoke a user-defined function as if it were a stored procedure. You
might want to do this to assign the return value to a variable. In this case, though, the syntax is different. You must not
specify the parameter list in parentheses, as in the examples above. Instead, you should just list them after the name of
the function:
DECLARE @avg money
EXEC @avg = dbo.AveragePrice 'business'
SELECT @avg

Additional Restrictions
The SQL statements inside your scalar-valued functions cannot include any nondeterministic system functions. I
talked about determinism in Chapter 8 when I discussed the requirements for creating Indexed Views and indexes on
computed columns.

For example, suppose I want a function that will format today's date using any separator that I supply. I might try to
write the function like this:
CREATE FUNCTION MyDateFormat
(@Separator char(1)='-')
RETURNS nchar(20)
AS
BEGIN
DECLARE @indate datetime
SELECT @indate = GETDATE()
RETURN
CONVERT(nvarchar(20), DATEPART(dd, @indate))
+ @Separator
+ CONVERT(nvarchar(20), DATEPART(mm, @indate))
+ @Separator
+ CONVERT(nvarchar(20), DATEPART(yy, @indate))
END

However, because GETDATE is nondeterministic, I get this error:

Server: Msg 443, Level 16, State 1, Procedure MyDateFormat, Line 7
Invalid use of 'getdate' within a function.

I can change the function definition to accept a datetime value as an input parameter and then call the function with
GETDATE as an argument. Not only does this allow me to accomplish my goal, but also it makes my function much
more versatile. Here's the function:
CREATE FUNCTION MyDateFormat
This document is created with the unregistered version of CHM2PDF Pilot

(@indate datetime,
@Separator char(1)='-')
RETURNS nchar(20)
AS
BEGIN
RETURN
CONVERT(nvarchar(20), DATEPART(dd, @indate))
+ @Separator
+ CONVERT(nvarchar(20), DATEPART(mm, @indate))
+ @Separator
+ CONVERT(nvarchar(20), DATEPART(yy, @indate))
END

And here's an example of its invocation:

SELECT dbo.MyDateFormat(GETDATE(), '*')

RESULT:
--------------------
18*7*2000

As a final example, I'll rewrite the factorial routine as a function. As you've seen, we can go only one value higher
using the iterative solution instead of the recursive solution. Since functions work very nicely recursively, I'll rewrite it
as a recursive function. The function will call itself by multiplying the input parameter by the factorial of the number one
less than the parameter. So 10 is computed by finding the factorial of 9 and multiplying by 10. To avoid overflow
problems, the function simply returns a 0 for the result of any input value that is out of range. Here's the function:
CREATE FUNCTION fn_factorial (@param decimal(38, 0) )
RETURNS decimal(38, 0)

AS
BEGIN
IF (@param < 0 OR @param > 32) RETURN (0)
RETURN (CASE
WHEN @param > 1 THEN @param * dbo.fn_factorial(@param - 1)
ELSE 1
END)
END

Unlike the factorial procedure, which lists all the factorial values up to and including the factorial of the input
parameter, the function fn_factorial simply returns the value that is the factorial for the argument. Remember that you
must always specify a two-part name when you call a user-defined scalar function:
SELECT factorial = dbo.fn_factorial(10)

RESULT:
factorial
----------------------------------------
3628800

Table-Valued Functions
Table-valued functions return a rowset. You can invoke them in the FROM clause of a SELECT statement, just as
you would a view. In fact, you can think of a table-valued function as if it were a parameterized (or parameterizable)
view. A table-valued function is indicated in its definition using the word TABLE in the RETURNS clause. There are
two ways to write table-valued function: as inline functions or as multistatement functions. This difference is relevant
This document is created with the unregistered version of CHM2PDF Pilot

only to the way the function is written; all table-valued functions are invoked in the same way. You'll also see in
Chapter 15 that there is a difference in the way that the query plans for inline and multistatement functions are cached.

Inline Functions
If the RETURNS clause specifies TABLE with no additional table definition information, the function is an inline
function and there should be a single SELECT statement as the body of the function. Here's a function that will return
the names and quantities of all the books sold by a particular store for which the store ID is passed as an input
parameter:
CREATE FUNCTION SalesByStore(@storid varchar(30))
RETURNS TABLE
AS
RETURN (SELECT title, qty
FROM sales s, titles t
WHERE s.stor_id = @storid AND t.title_id = s.title_id)

We can execute this function by invoking it in a FROM clause:

SELECT * FROM SalesByStore ('8042')

RESULT:
title qty
------------------------------------------------------ ------
The Gourmet Microwave 15
The Busy Executive's Database Guide 10
Cooking with Computers: Surreptitious Balance Sheets 25
But Is It User Friendly? 30

Note that for a table-valued function, you don't have to include the owner name.

Multistatement Table-Valued Functions

If the RETURNS clause specifies that it's returning a TABLE and specifying a table variable name, the function is a
multistatement table-valued function. The RETURNS clause also lists the columns and datatypes for the table. The
body of the function populates the table using INSERT and possibly UPDATE statements. When the RETURN is
executed, it simply returns the table variable specified earlier in the RETURNS clause.

Here's an example of the previous inline function rewritten as a multistatement function:

CREATE FUNCTION SalesByStore_MS(@storid varchar(30))
RETURNS @sales TABLE(title varchar(80), qty int)
AS
BEGIN
INSERT @sales
SELECT title, qty
FROM sales s, titles t
WHERE s.stor_id = @storid AND t.title_id = s.title_id
RETURN
END

The function is invoked exactly as the inline table-valued function is invoked.

The following statements are the only ones allowed in a multistatement table-valued function:
 Assignment statements
 Control-of-flow statements
This document is created with the unregistered version of CHM2PDF Pilot

 DECLARE statements that define data variables and cursors that are local to the function
 SELECT statements containing select lists with expressions that assign values to variables that are local to the
function
 Cursor operations referencing local cursors that are declared, opened, closed, and deallocated in the function.
Only FETCH statements that assign values to local variables using the INTO clause are allowed; FETCH
statements that return data to the client are not allowed. (For more details on cursors, see Chapter 13.)
 INSERT, UPDATE, and DELETE statements that modify table variables that are local to the function

Also, like for scalar functions, table-valued functions cannot contain any built-in nondeterministic functions.

Side Effects
Basically, the statements that aren't allowed are ones that return data other than the function's return value and the
ones that product side effects. A side effect is a change to some persisted state that is not local to the function.
Invoking a function should not change your database in any way; it should only return a value (scalar or table-valued)
to the client. Thus, the following are not allowed:
 Updates to tables
 Global cursor statements
 Creation of objects
 Transaction control statements

System Table-Valued Functions

SQL Server 2000 includes a set of systemwide table-valued functions that can be invoked from any database.
Normally, the table-valued functions you create will be objects in one particular database, and to invoke them from
another database, you must specify the database name. System table-valued functions do not have this restriction.
However, to invoke one of these functions, you must use a special syntax and precede the function name with ::. In
fact, you cannot specify an owner or database name; you can only use this special syntax.

Here are the documented system table-valued functions:

fn_helpcollations
fn_listextendedproperty
fn_servershareddrives
fn_trace_geteventinfo
fn_trace_getfilterinfo
fn_trace_getinfo
fn_trace_gettable
fn_virtualfilestats
fn_virtualservernodes

The function fn_virtualfilestats returns I/O statistics for a database file and requires a database ID and a file ID as
parameters. The following example calls this function for the data file of the pubs database. We'll discuss the meaning
of some of the return values in Chapter 17.
SELECT * FROM ::fn_virtualfilestats(5, 1)

RESULT:
DbId FileId TimeStamp NumberReads NumberWrites BytesRead
------ ------ ----------- ------------ ------------ ------------
5 1 34138899 115 9 942080

BytesWritten IoStallMS
------------- -----------------
90112 4585
This document is created with the unregistered version of CHM2PDF Pilot

Managing User-Defined Functions

Just like a view, a function can be created WITH SCHEMABINDING. This option means that any object
referenced by the function cannot be dropped or altered. Any attempt to alter such a referenced object will result in
an error. Let's use the inline table-value function I created above and re-create it WITH SCHEMABINDING. Note
that we can use ALTER FUNCTION to change the function's definition without dropping and re-creating it.
However, we cannot use ALTER to change a table-valued function to a scalar function or to change an inline function
to a multistatement function.
ALTER FUNCTION SalesByStore(@storid varchar(30))
RETURNS TABLE
WITH SCHEMABINDING
AS
RETURN (SELECT title, qty
FROM dbo.sales s, dbo.titles t
WHERE s.stor_id = @storid AND t.title_id = s.title_id)

A function can be schema-bound only if the following conditions are true:

 The user-defined functions and views referenced by the function are also schema-bound.
 All objects referenced by the function are referred to by a two-part name.
 The function is in the same database as all the referenced objects.
 Any user referencing the function has REFERENCES permission on all the database objects that the function
references.

SQL Server keeps track of objects that are dependent on other objects in the sysdepends table. The data in the table
looks very cryptic—it's all numbers. To get meaningful information from the sysdepends table, you have to get the
names of the objects stored in the id column, which are the dependent objects, and the names of the objects stored in
the depid column, which are the referenced objects.

For example, in my SalesByStore function above, the function itself is the dependent object, and the objects it
depends on are the titles and sales tables. The depnumber column refers to a column in the referenced object that
the dependent object depends on. My SalesByStore functions depend on the qty, stor_id, and title_id columns
from sales and the title_id and title columns from titles. Finally, the column deptype indicates whether the
dependency is schema-bound—that is, whether a change to the referenced column should be prevented. By
accessing the syscolumns and sysdepends tables and using the OBJECT_NAME function to extract the name from
sysobjects, we can see what columns are schema-bound in the SalesByStore functions:
SELECT obj_name = SUBSTRING(OBJECT_NAME(d.id), 1, 20),
dep_obj = SUBSTRING(OBJECT_NAME(d.depid), 1, 20),
col_name = SUBSTRING(name, 1, 15),
IsSchemaBound = CASE deptype
WHEN 1 THEN 'Schema Bound'
ELSE 'Free'
END
FROM
sysdepends d
JOIN syscolumns c ON d.depid = c.id
AND d.depnumber = c.colid
WHERE object_name(d.id) LIKE 'SalesByStore%'

RESULTS:
obj_name dep_obj col_name IsSchemaBound
-------------------- -------------------- --------------- -------------
SalesByStore sales stor_id Schema Bound
SalesByStore sales qty Schema Bound
This document is created with the unregistered version of CHM2PDF Pilot

SalesByStore sales title_id Schema Bound

SalesByStore titles title_id Schema Bound
SalesByStore titles title Schema Bound
SalesByStore_MS sales stor_id Free
SalesByStore_MS sales qty Free
SalesByStore_MS sales title_id Free
SalesByStore_MS titles title_id Free
SalesByStore_MS titles title Free

To see the impact of schema binding, I'll try to change one of the referenced columns. The following ALTER will
attempt to change the datatype of qty from smallint to int:
ALTER TABLE sales
ALTER COLUMN qty int

I get these error messages:

Server: Msg 5074, Level 16, State 3, Line 1
The object 'SalesByStore' is dependent on column 'qty'.
Server: Msg 4922, Level 16, State 1, Line 1
ALTER TABLE ALTER COLUMN qty failed because one or more objects
access this column.

On the other hand, changing the datatype of the ord_date column from datetime to smalldatetime will succeed
because the ord_date column is not referenced by a schema-bound object.

Getting Information About Your Functions

SQL Server 2000 provides many utilities for getting information about your functions. To see the definition of a
function, you can use the system procedure sp_helptext or look in the ANSI SCHEMA VIEW called ROUTINES.
Both commands shown below will return the definition of my SalesByStore function:
EXEC sp_helptext SalesByStore

SELECT routine_definition
FROM INFORMATION_SCHEMA.routines
WHERE routine_name = 'SalesByStore'

Although you can use either sp_helptext or ROUTINES to get information about your own functions, you can't get
the definition of the system table-valued functions using sp_helptext. You must use the master database and look in
the ROUTINES view. In fact, for some of the supplied system table-valued functions, you won't get the entire
definition. You'll get only the declaration section because the rest of the code is hidden. The query below shows the
complete definition of a system function:
USE master
SELECT routine_definition
FROM INFORMATION_SCHEMA.ROUTINES
WHERE routine_name = 'fn_helpcollations'

You can also get information about your functions from the ANSI SCHEMA VIEWS called
ROUTINE_COLUMNS and PARAMETERS. For details about using these views, see SQL Server Books Online.

The metadata function OBJECTPROPERTY also has quite a few property values that are relevant to user-defined
functions. For example, I mentioned earlier that you can tell whether a function is a table-valued function by looking at
This document is created with the unregistered version of CHM2PDF Pilot

its definition and checking to see whether the RESULTS clause contains the word TABLE. Alternatively, you can use
the OBJECTPROPERTY function with the IsTableFunction or 'IsInlineFunction' argument:
SELECT OBJECTPROPERTY(object_id('SalesByStore'), 'IsInlineFunction')

Calling the OBJECTPROPERTY function with the 'IsInlineFunction' property parameter returns one of three values.
The value 1 means TRUE, the function is an inline table-valued function and 0 means FALSE, the function isn't an
inline table-valued function. A NULL will be returned if you typed something wrong, for example, if you supply an
invalid object ID, the ID of an object that is not a function, or if you misspell the property name. If you call the
OBJECTPROPERTY function with the 'IsTableFunction' property parameter, it returns a 1 if the function is a
table-valued function that is not also an inline function. Here are the other parameters of OBJECTPROPERTY that
can give you useful information about your functions:
IsScalarFunction
IsSchemaBound
IsDeterministic
This document is created with the unregistered version of CHM2PDF Pilot

34

Rewriting Stored Procedures as

Functions
User-defined functions are new in SQL Server 2000. In previous releases, many applications had stored procedures
to perform tasks that could have been much more naturally coded as functions, but because there was no facility for
writing user-defined functions, stored procedures had to be used. If you've upgraded to SQL Server 2000 from an
earlier release, you might be wondering which of your existing stored procedures you should rewrite as functions.

The following list of criteria for creating functions from stored procedures mostly duplicates what you'll find in SQL
Server Books Online, but I've included it here for convenience. Your main consideration should be whether you want
to be able to invoke an existing stored procedure directly from within a query. If so, you should consider re-creating
the procedures as a function.

In general, if the stored procedure returns a (single) result set, you should define a table-valued function. If the stored
procedure computes a scalar value, you should define a scalar function.

If a stored procedure meets the following criteria, it's a good candidate for being rewritten as a table-valued function:
 The logic is expressible in a single SELECT statement but is a stored procedure, rather than a view, only
because of the need for parameters. You can handle this scenario using an inline table-valued function.
 The stored procedure does not perform update operations (except to table variables).
 There is no need for dynamic EXECUTE statements (discussed later in this chapter)
 The stored procedure returns one result set.
 The primary purpose of the stored procedure is to build intermediate results that are to be loaded into a
temporary table, which is then queried in a SELECT statement.
This document is created with the unregistered version of CHM2PDF Pilot

34

Rolling Your Own System

Routines
You've seen that SQL Server 2000 provides both system-level stored procedures and system-level table-valued
functions. The two types of objects are similar in that they can be invoked from any database even though they are
stored only in the master database.

Your Own System Procedures

All the system stored procedure names begin with sp_. This is more than just a convention. Any procedure created in
the master database that begins with sp_ can be called from any other database without having to be fully referenced
with the database name. This can also be useful for procedures you create. The sp_ magic works even for extended
stored procedures, which are user-written calls to DLLs. By convention, extended stored procedure names begin
with xp_, but the sp_ prefix and its special property can be applied to them as well (but only when added to the
master database). In fact, some extended procedures that are supplied as part of the product, such as those used to
create Automation objects (for example, sp_OACreate), use the sp_ prefix so that they can be called from anywhere,
even though they're actually functions in a DLL, not a Transact-SQL stored procedure.

The sp_ prefix actually makes a procedure special in two ways. First, as I've mentioned, a procedure whose name
starts with sp_ can be called directly from any database, without a fully qualified name. Second, system tables
referenced in your special procedures will always refer to the tables in the database from which the procedure was
called. For example, the sp_help stored procedure lists all the objects from the sysobjects system table. But every
database has its own sysobjects table, so which one is used? If you execute sp_help from the pubs database, you get
a list of the objects in pubs; if you call sp_help from msdb, you get a list of objects in msdb, and so on. Two tables
are exceptions to this rule. For some undetermined reason, if your system stored procedure references the sysfiles or
sysfilegroups table, the procedure will always access the table from the master database even though there is a
sysfiles and sysfilegroups table in every database.

The following example illustrates this behavior:

CREATE PROC sp_list_files_and_objects
AS
SELECT filename FROM sysfiles
SELECT name from sysobjects
RETURN

I won't show you the output from executing the procedure because the list of objects will probably be quite long no
matter what database you run this procedure from. But you can try it yourself:
USE pubs
EXEC sp_list_files_and_objects
USE northwind
EXEC sp_list_files_and_objects

You'll always see the same list of filenames, which are the ones for the master database, but the list of objects will
change depending on your current database.
This document is created with the unregistered version of CHM2PDF Pilot

Of course, if the sp_ procedure references a table that exists only in the master database, such as sysconfigures, the
table in master will be referenced no matter where you are when you call the procedure. This trick works only for
system tables, however. If you create a user table called MasterLookup and reference it in an sp_ stored procedure,
SQL Server will look only in the master database to try to find the MasterLookup table. If it doesn't exist in master,
you'll get an error message when executing your procedure, even if you call the procedure from the database where
the table is located.
WARNING
Microsoft strongly discourages any direct references to the system tables and suggests that you instead use only the
presupplied stored procedures and the object property functions to get any system information you need. It's not
guaranteed that the structures of the system tables will remain the same from one release to the next, or even between
service packs. In fact, the Upgrade Wizard for converting SQL Server 6 to SQL Server 2000 doesn't even attempt
to upgrade any procedures that modify system tables.

I don't recommend that you use sp_ as a prefix for your own local stored procedures. The sp_ prefix has a special
meaning, and unless you intend your procedure to have this special meaning and exist in the master database, it can be
confusing for users who are aware of the special meaning.

Your Own System Functions

SQL Server 2000 also provides system functions that are treated in some ways just like the user-defined functions
that you can write. These are the functions that are listed in the sysobjects table and that are listed when you select
from the ROUTINES view. These system functions are different from the ones that you would normally create
because they are available from any database. You saw the list of these functions, but I'll repeat it here:
fn_helpcollations
fn_listextendedproperty
fn_servershareddrives
fn_trace_geteventinfo
fn_trace_getinfo
fn_virtualfilestats
fn_virtualservernodes

Unlike the prefix for system stored procedures, the prefix fn_ by itself has no special meaning. In fact, I used it when
creating my factorial function to distinguish my function from my procedure. In addition, it's not nearly as easy to
create system functions of your own. Although it is possible, it is not supported at all, and you need to be aware that
any system functions you write in one release might not work in a subsequent release.

The supplied system functions listed above all have the following characteristics:
 They are objects in the master database.
 Their names start with fn_.
 They are owned by the user system_function_schema.
 Their names use only lowercase letters.

If you want your function to be available from any database, it must have the characteristics listed above. However,
it's not easy to create a function owned by system_function_schema. If you try, you get an error that there is no such
user, even though that value is listed as a name in the sysusers table.

In order to create objects owned by the user name system_function_schema, you must enable updates to the
system tables, which is why this technique is unsupported and undocumented. In addition, you cannot create a
recursive system function. So, to create our factorial function as a system function, you have to use the iterative
approach. Here's a complete example of creating my own system function:
USE master
GO
This document is created with the unregistered version of CHM2PDF Pilot

EXEC sp_configure 'allow updates', 1

GO

RECONFIGURE WITH OVERRIDE

GO

CREATE FUNCTION system_function_schema.fn_factorial (@param1 int)

RETURNS NUMERIC(38, 0)
AS
BEGIN
DECLARE @counter int, @result NUMERIC(38, 0)
IF (@param1 < 0 OR @param1 > 33) RETURN (0)

SET @counter=1 SET @result=1

WHILE (@counter < @param1 AND @param1 <> 0 )

BEGIN
SET @result=@result * (@counter + 1)
SET @counter=@counter + 1
END
RETURN (@result)
END
GO

EXEC sp_configure 'allow updates', 0

GO

RECONFIGURE WITH OVERRIDE

GO

Now invoke it:

USE <some_other_db>
GO

SELECT fn_factorial(5)
This document is created with the unregistered version of CHM2PDF Pilot

34

Executing Batches, or What's

Stored About Stored Procedures
(and Functions)?
Typically, when a batch of Transact-SQL commands is received from a client connection, the following high-level
steps are performed:

Step One: Parse Commands and Create the

Sequence Tree
The command parser checks for proper syntax and translates the Transact-SQL commands into an internal format
that can be operated on. The internal format is known as a sequence tree or a query tree. The command parser
handles these language events.

Step Two: Compile the Batch

An execution plan is generated from the sequence tree. The entire batch is compiled, queries are optimized, and
security is checked. The execution plan contains the necessary steps to check any constraints that exist. If an after
trigger exists for any data modification statement, the call to that trigger is appended to the execution plan for the
statement. If an instead-of trigger exists, the call to that trigger replaces the execution plan for the data modification
statement. Recall that a trigger is really a specialized type of stored procedure. Its plan is cached, and the trigger
doesn't need to be recompiled every time it is invoked.

The execution plan includes the following:

 All the necessary steps to carry out the commands in the batch or stored procedure.
 The steps needed to enforce constraints. (For example, for a foreign key, this would involve checking values
in another table.)
 A branch to the stored procedure plan for a trigger, if one exists.

Step Three: Execute

During execution, each step of the execution plan is dispatched serially to a manager that's responsible for carrying
out that type of command. For example, a data definition command (in DDL), such as CREATE TABLE, is
dispatched to the DDL manager. DML statements, such as SELECT, UPDATE, INSERT, and DELETE, go to the
DML manager. Miscellaneous commands, such as DBCC and WAITFOR, go to the utility manager. Calls to stored
procedures (for example, EXEC sp_who) are dispatched to the stored procedure manager. A statement with an
explicit BEGIN TRAN interacts directly with the transaction manager.

Contrary to what you might think, the execution plans for stored procedures and functions are not permanently stored
on disk. (This is a feature—the execution plan is relatively dynamic.) Think for a moment about why it's important for
the execution plan to be dynamic. As new indexes are added, preexisting indexes are dropped, constraints are added
or changed, and triggers are added or changed; or as the amount of data changes, the plan can easily become
This document is created with the unregistered version of CHM2PDF Pilot

obsolete.

So, what's stored about a stored procedure or function?

The SQL statements that were used to create the routine are stored in the system table syscomments. The first time
a routine is executed after SQL Server was last restarted, the SQL text is retrieved and an execution plan is compiled.
The execution plan is then cached in SQL Server's memory, and it remains there for possible reuse until it's forced out
in a least recently used (LRU) manner. We'll look at more details about the SQL Server's caching mechanism and
when plans are forced out of cache in Chapter 15.

Hence, a subsequent execution of the routine can skip not only step 1, parsing, but also step 2, compiling, and go
directly to step 3, execution. Steps 1 and 2 always add some overhead and can sometimes be as costly as actually
executing the commands. Obviously, if you can eliminate the first two steps in a three-step process, you've done well.
That's what the stored routines let you do. Note that most of this discussion of routines applies to triggers as well as to
stored procedures and functions, even though triggers are not listed in the SCHEMA VIEW called ROUTINES.

When you execute a routine, if a valid execution plan exists in the procedure cache, it will be used (eliminating the
parsing and compiling steps). When the server is restarted, no execution plans will be in the cache; so the first time the
server is restarted, routines will all be compiled the first time they are executed.
TIP
You can preload your procedure cache with execution plans for routines by defining a startup stored procedure that
executes the routines you want to have compiled and cached.

After a routine executes, its plan remains in the cache and is reused the next time any connection executes the same
routine. In addition, because multiple connections might try to execute the same plan concurrently, a part of the plan
(called the shareable portion) is reentrant. Each concurrent user of that plan also requires an execution context. If you
think of this in traditional programming terms, this execution context is needed to contain the dirty data.

The execution context tends to be fairly small compared to the plans. The execution contexts are themselves serially
reusable for certain plans that don't contain special operators. Figures 11-1 and 11-2 show execution with and
without a stored routine.

Figure 11-1. Efficient execution using a stored routine.

Figure 11-2. Less efficient execution with an ad hoc query.

In Chapter 15, you'll see that even ad hoc queries do not need to be recompiled every time they're run. But when
and how their plans are stored and reused is not nearly as predictable as with stored routines. In addition, you have
little explicit control over the recompilation of ad hoc queries.

Step Four: Recompile Execution Plans

This document is created with the unregistered version of CHM2PDF Pilot

By now, it should be clear that the SQL code for routines persists in the database but execution plans don't.
Execution plans are cached in memory. But sometimes they can be invalidated and a new plan generated.

So, when is a new execution plan compiled?

The short answer is: whenever SQL Server needs to! This can include the following cases:
 When a copy of the execution plan isn't available in memory.
 When an index on a referenced object is dropped.
 When updated statistics are available for any table used by the routine. (Chapter 15 covers statistics in
detail.)
 When an object referenced in the routine is altered using ALTER TABLE.
 When the object has been named as the argument to sp_recompile. If the object is a routine, the routine will
be marked so that it is recompiled the next time it is run. If the object is a table or view, all the routines that
reference that object will be recompiled the next time they are run. The system procedure sp_recompile
increments the schema column of sysobjects for a given object. This invalidates any plans for the object as
well as any plans that reference the object, as described in the previous examples.
 When a stored procedure has been created using the WITH RECOMPILE option. A stored procedure can
be created using WITH RECOMPILE to ensure that its execution plan will be recompiled for every call and
will never be reused. Note that the WITH RECOMPILE option is available only when you create stored
procedures and is not available for function creation. Creating a stored procedure WITH RECOMPILE can
be useful if procedures take parameters and the values of the parameters differ widely, resulting in a need for
different execution plans to be formulated. For example, if a procedure is passed a value to be matched in the
WHERE clause of a query, the best way to carry out that query can depend on the value passed. SQL
Server maintains statistics for each index as a histogram to help it decide whether the index is selective enough
to be useful.

For one given value, an index might be highly selective, and the distribution statistics might indicate that only 5
percent of the rows have that value. SQL Server might decide to use that index because it would exclude
many pages of data from having to be visited. Using the index would be a good strategy, and the cached plan
for the procedure might include the instructions to use the index. However, using another value, the index
might not be so selective. Rather than use only the index to visit most of the data pages (ultimately doing more
I/O because you're reading both the index and the data), you'd be better off simply scanning the data and not
reading the index.

For example, suppose we have a nonclustered index on the color column of our automobile table. Forty
percent of the cars are blue, 40 percent are red, and 5 percent each are yellow, orange, green, and purple.
It's likely that a query based on color should scan the table if the color being searched on is blue or red, but it
should use the index for the other colors. Without using the WITH RECOMPILE option, the execution plan
created and saved would be based on the color value the first time the procedure was executed.

So if we passed yellow to the procedure the first time it executed, we'd get a plan that used an index.
Subsequently, if we passed blue, we might be able to use the previous plan that was created for yellow. In
this case, however, we'd be traversing a nonclustered index to access a large percentage of rows in the table.
This could end up being far more expensive than simply scanning the entire table. In such a case, when there's
a lot of variance in the distribution of data and execution plans are based on the parameters passed, it makes
sense to use the WITH RECOMPILE option.

This example also shows that two execution plans for the same procedure can be different. Suppose we're
not using WITH RECOMPILE, and we execute the procedure for both blue and green simultaneously from
two different connections. Assume for a moment that no plan is cached. Each will generate a new plan, but
one plan will use the index and the other won't. When a subsequent request for red arrives, the plan it uses
will be a matter of chance. And if two simultaneous calls come in for red and each plan is available, the two
equivalent requests will execute differently because they'll use different plans. If, as you're processing queries,
you see significant deviations in the execution times of apparently identical procedures, think back to this
example.
This document is created with the unregistered version of CHM2PDF Pilot

 When the stored procedure or function is executed using the WITH RECOMPILE option. This case is
similar to the preceding one, except that the routine isn't created with the option; rather, the option is specified
when the routine is invoked. The WITH RECOMPILE option can always be added upon execution, forcing a
new execution plan to be generated. Note that you can use this option with user-defined scalar functions, but
only if you invoke the function with the EXEC keyword.

Storage of Routines
With each new routine, a row is created in the sysobjects table, as happens for all database objects. The text of a
routine (including comments) is stored in syscomments, which is typically useful. Storing the text allows procedures
such as sp_helptext to display the source code of a routine so that you can understand what's going on, and it allows
function editors and debuggers to exist.

Most users and developers find it helpful to have the full text of a routine stored in clear text. There is a limit on the
size of the text for a routine due to the amount of text that syscomments can store for a given routine, but you're not
likely to run up against that limit soon. The syscomments.text field can hold up to 8000 bytes, and you can have
multiple rows in syscomments for each routine. The sequence of the rows for a given routine is tracked in the colid
column. Since colid is a smallint, there can be up to 32,767 rows for any routine. Any one routine should then be
able hold 8000 × 32,767 bytes, or 250 MB. There's one additional limiting factor, however: the maximum size of a
batch. To create the routine, the client must send the complete definition to SQL Server in the CREATE statement,
and this must be sent as a single batch. SQL Server has a limit on the batch size, which is 65,536 times the network
packet size. Because the default value for network packet size is 4096 bytes, we get a batch size of 256 MB. So the
limit on the size of your routines text isn't worth worrying about right away.

Encrypting Routines
With the rollout of version 6, the SQL Server developers at Microsoft learned somewhat painfully that some users
didn't appreciate that the text of certain programmable objects was available in the syscomments system table.
Several ISVs had built integrated solutions or tools that created stored procedures to use with earlier versions of SQL
Server. In most cases, these solutions were sophisticated applications, and the ISVs viewed the source code as their
proprietary intellectual property. They noticed that the text of the procedure in syscomments didn't seem to do
anything, so they set the text field to NULL and the procedure still ran fine. In this way, they avoided publishing their
procedure source code with their applications.

Unfortunately, when it came time to upgrade a database to version 6, this approach created a significant problem.
The internal data structures for the sequence plans had changed between versions 4.2 and 6. The ISVs had to
re-create procedures, triggers, and views to generate the new structure. Although SQL Server's Setup program was
designed to do this automatically, it accomplished the tasks by simply extracting the text of the procedure from
syscomments and then dropping and re-creating the procedure using the extracted text. Obviously, this automatic
approach failed for procedures in which the creator had deleted the text.

When the SQL Server developers at Microsoft learned of this problem after the release of the version 6 beta, they
immediately understood why developers had felt compelled to delete the text. Nonetheless, they couldn't undo the
work that they had already done. Developers with these ISVs had to dig out their original stored procedure creation
scripts and manually drop and re-create all their procedures. It might have been possible to create a converter
program that would operate purely on the internal data structures used to represent procedures and views, but it
wasn't practical. Attempting this would have been like developing a utility to run against an executable program and
have it reverse-engineer the precise source code (more than a disassembly) that was used to create the binary. The
SQL source code compilation process is designed to be a descriptive process that produces the executable, not an
equation that can be solved for either side.

After the beta release but before the final release of version 6, the developers added the ability to encrypt the text
stored in syscomments for stored procedures, triggers, and views. This allowed programmers to protect their source
This document is created with the unregistered version of CHM2PDF Pilot

code without making it impossible for the upgrade process to re-create stored procedures, triggers, and views in the
future. You can now protect your source code by simply adding the modifier WITH ENCRYPTION to CREATE
PROCEDURE. No decrypt function is exposed (which would defeat the purpose of hiding the textlike source code).
Internally, SQL Server can read this encrypted text and upgrade the sequence trees when necessary. Because the text
isn't used at runtime, no performance penalty is associated with executing procedures created using WITH
ENCRYPTION.
NOTE
You give up some capabilities when you use WITH ENCRYPTION. For example, you can no longer use the
sp_helptext stored procedure or object editors that display and edit the text of the stored procedure, and you can't
use a source-level debugger for Transact-SQL. Unless you're concerned about someone seeing your procedures, you
shouldn't use the WITH ENCRYPTION option.

With SQL Server 2000, if you create a procedure, function, trigger, or view using WITH ENCRYPTION, the status
column of syscomments will have a value of 1 (it's first bit will be set) and if it's not encrypted, the status column will
be 2 (the second bit will be set). The encrypted and texttype columns of syscomments will also reflect the
encryption status, but these columns are both computed columns based on the value that is in the status column. The
value of texttype is 6 for an encrypted routine and 2 otherwise. If you want to programmatically determine whether a
routine is encrypted, it's safer to check the encrypted column for a value of 0 or 1. Also, the SCHEMA VIEW called
ROUTINES will have NULL in the routine_definition column if the routine is encrypted.

I created two procedures—one encrypted and one not—to illustrate the effects on syscomments:
CREATE PROCEDURE cleartext
AS
SELECT * FROM authors
GO

CREATE PROCEDURE hidetext WITH ENCRYPTION

AS
SELECT * FROM authors
GO

SELECT o.name, o.id,

number, colid, c.status, texttype, encrypted, text
FROM syscomments c JOIN sysobjects o
ON o.id=c.id
WHERE o.id=OBJECT_ID('hidetext') OR o.id=OBJECT_ID('cleartext')

Here's the output:

name id number colid status texttype encrypted
------------- ----------- ------ ------ ------ -------- ---------
cleartext 1189579276 1 1 2 2 0

hidetext 1205579333 1 1 1 6 1

text
--------------------------------
CREATE PROCEDURE cleartext
AS
SELECT * FROM authors

???????l???????????????????????

To find created objects that have encrypted text, you can use a simple query:
This document is created with the unregistered version of CHM2PDF Pilot

-- Find the names and types of objects that have encrypted text
SELECT name, type FROM syscomments c JOIN sysobjects o
ON o.id=c.id
WHERE encrypted = 1

Here's the output:

name type
--------------------- ----
hidetext P

The values that you might see in the type column are P (procedure), TR (trigger), V (view), FN (scalar function), and
TN (table function). If you try to run sp_helptext against an encrypted procedure, it will return a message stating that
the text is encrypted and can't be displayed:
EXEC sp_helptext 'hidetext'
The object's comments have been encrypted.

Altering a Routine
SQL Server allows you to alter the definition of a routine. The syntax is almost identical to the syntax for creating the
routine initially, except that the keyword CREATE is replaced by the keyword ALTER.

The big difference is that the routine name used with ALTER must already exist, and the definition specified replaces
whatever definition the routine had before. Just like when you use the ALTER VIEW command, the benefit of
ALTER PROCEDURE or ALTER FUNCTION comes from the fact that the routine's object_id won't change, so
all the internal references to this routine will stay intact. If other routines reference this one, they'll be unaffected. If
you've assigned permissions on this routine to various users and roles, dropping the routine removes all permission
information. Altering the routine keeps the permissions intact.
This document is created with the unregistered version of CHM2PDF Pilot

34

Temporary Stored Procedures

Temporary stored procedures allow an execution plan to be cached, but the object's metadata and the text of the
procedure are stored in the system tables of the tempdb database—in sysobjects and syscomments. Recall that
tempdb is re-created every time SQL Server is restarted, so these objects no longer exist after SQL Server is shut
down. During a given SQL Server session, you can reuse the procedure without permanently storing it.

Typically, you use a temporary stored procedure when you want to regularly execute the same task several times in a
session, although you might use different parameter values and you don't want to permanently store the task. You
could conceivably use a permanent stored procedure and drop it when you're finished, but you'd inevitably run into
cleanup issues if a stored procedure were still hanging around and the client application terminated without dropping
the procedure. Because temporary stored procedures are deleted automatically when SQL Server is shut down (and
tempdb is created anew at startup), cleanup isn't an issue.

Just as SQL Server has three types of temporary tables, it also has three types of temporary stored procedures:
private, global, and those created from direct use of tempdb.

Private Temporary Stored Procedures

By adding a single pound sign (#) to the beginning of the stored procedure name (as in CREATE PROC
#get_author AS…), you can create the procedure from within any database as a private temporary stored
procedure. Only the connection that created the procedure can execute it, and you can't grant privileges on it to
another connection. The procedure exists for the life of the creating connection only; that connection can explicitly use
DROP PROCEDURE on it to clean up sooner. Because the scoping of a private temporary procedure is specific
only to the connection that created it, you won't encounter a name collision if you choose a procedure name that's
used by another connection. As with temporary tables, you use your private version, and what occurs in other
connections is irrelevant.

Global Temporary Stored Procedures

By adding two pound signs (##) to the beginning of the stored procedure name (as in CREATE PROC
##get_author AS…), you can create the procedure from within any database as a global temporary stored
procedure. Any connection can subsequently execute that procedure without EXECUTE permission being specifically
granted. Unlike with private temporary stored procedures, only one copy of a global temporary stored procedure
exists for all connections. If another connection has created a procedure with the same name, the two names will
collide and the create procedure statement will fail. Permission to execute global temporary procedures defaults to
public and can't be changed. You can issue a command to deny other users permission, but it will have no effect. Any
user, on any connection, can execute a global temporary stored procedure.

A global temporary stored procedure exists until the creating connection terminates and all current execution of the
procedure completes. Once the creating connection terminates, however; no further execution is allowed. Only the
connections that have already started executing are allowed to finish.

Procedures Created from Direct Use of tempdb

Realizing that tempdb is re-created every time SQL Server is started, you can create a procedure in tempdb that
fully qualifies objects in other databases. Procedures created in tempdb in this way can exist even after the creating
This document is created with the unregistered version of CHM2PDF Pilot

connection is terminated, and the creator can specifically grant and deny execute permissions to specific users. To do
this, the creator of the procedure must have create procedure privileges in tempdb. You can set up privileges in
tempdb in one of two ways: you can set privileges in model (the template database) so that they will be copied to
tempdb when it is created at system restart, or you can set up an autostart procedure to set the tempdb privileges
every time SQL Server is started.

Here's an example of creating a procedure in tempdb and then executing it from the pubs database:
USE tempdb
GO

CREATE PROC testit AS

SELECT * FROM pubs.dbo.authors
GO

-- Executing the procedure created above from the pubs database

USE pubs
EXEC tempdb..testit

While we're on the subject of temporary objects, keep in mind that a private temporary table created within a stored
procedure isn't visible to the connection after the creating procedure completes. You can, however, create a local
temporary table before executing a stored procedure and make the table visible to the stored procedure. The scoping
of the temporary table extends to the current statement block and all subordinate levels.
NOTE
You can use the @@nestlevel system function to check for the visibility of temporary tables. A temporary table
created at nest level 0 will be visible to all further levels on that connection. A table created within a procedure at nest
level 1, for example, won't be visible when execution returns to the calling block at nest level 0. A global temporary
table, or a table directly created in tempdb without using either # or ##, will be visible no matter what the nesting
level.
This document is created with the unregistered version of CHM2PDF Pilot

34

Autostart Stored Procedures

SQL Server has the ability to mark a stored procedure as autostart. Autostart stored procedures are useful if you
want to perform housekeeping functions regularly or if you have a background daemon procedure that's always
expected to be running. Another handy use for an autostart procedure is to have it assign some privileges in tempdb.
Or the procedure can create a global temporary table and then sleep indefinitely using WAITFOR. This will ensure
that such a temporary table will always exist, because the calling process is the first thing executed and it never
terminates.

You can use the system stored procedure sp_procoption to make a stored procedure start automatically. This
procedure allows you to turn options for stored procedures on or off. In SQL Server 2000, the only available option
is startup. Here's the syntax of the command to enable a procedure to automatically execute on SQL Server startup:

sp_procoption procedure_name, startup, true

You can remove the startup option by executing the same procedure and changing the value to FALSE. A
procedure that's autostarted runs in the context of a system administrator account. (The procedure can use SETUSER
to impersonate another account.) A procedure with the startup option set to TRUE must be in the master database
and must be owned by the database owner (dbo) in master. You can, of course, reference objects in other databases
from within the startup procedure or even call procedures that exist in other databases. A startup procedure is
launched asynchronously, and it can execute in a loop for the entire duration of the SQL Server process. This allows
several such procedures to be launched simultaneously at startup. When a startup procedure is running, it's seen as an
active user connection.

A single startup procedure can nest calls to other stored procedures, consuming only a single user connection. Such
execution of the nested procedures is synchronous, as would normally be the case. (That is, execution in the calling
procedure doesn't continue until the procedure being called completes.) Typically, a stored procedure that's
autostarted won't generate a lot of output. Errors, including those raised with RAISERROR, will be written to the
SQL Server error log, and any result sets generated will seemingly vanish. If you need the stored procedure to return
result sets, you should use a stored procedure that calls the main stored procedure with INSERT/EXEC to insert the
results into a table.

If you want to prevent a procedure with the startup option from executing, you can start the server using trace flag
4022 or as a minimally configured server using the -f switch to sqlservr.exe. You can set the -T4022 flag by using the
SQL Server Properties dialog box in SQL Server Enterprise Manager. (Right-click on the name of your server and
choose Properties to get to this dialog box.) These safeguards allow you to recover from problems. Consider the
illustrative but perhaps absurd example of someone including a procedure that executes the SHUTDOWN command.
If you had given such a procedure the autostart option, SQL Server would immediately shut itself down before you
could do anything about it!
This document is created with the unregistered version of CHM2PDF Pilot

34

System Stored Procedures

SQL Server installs a large number of system stored procedures that are used mostly for administrative and
informational purposes. In many cases, these are called behind the scenes by the SQL-DMO objects used by SQL
Server Enterprise Manager and other applications. But the system stored procedures can also be called directly; only
a few years ago, doing so was the primary mechanism by which SQL Server was administered. Old-time SQL
Server users were indoctrinated into using system stored procedures.

With the great tools and interfaces that are a core part of SQL Server today, there's no longer much reason to work
with these system stored procedures directly. But it's good to be familiar with them—understanding them can help you
understand the operations that occur on the system tables and can take much of the mystery out of what's going on
behind the scenes with the graphical tools. If you look carefully through the sysobjects system table (particular in the
master database), you'll find system procedures and system functions that are not among the documented system
routines. Typically, these routines exist to be called by some other system routine that is exposed; to support some
SQL Server utility, such as Enterprise Manager; or to provide statistics to the Microsoft System Monitor. These
procedures aren't documented for direct use because they exist only to support functionality exposed
elsewhere—they don't provide that functionality independently.

There's nothing secret about these routines. For the most part, their text is exposed clearly in syscomments. You're
welcome to explore them to see what they do and use them if you want. But unlike with the documented system
routines, Microsoft has no commitment to maintaining the undocumented system routines or making them exhibit
exactly consistent behavior in future releases. Of course, if your applications were to become dependent on one of
these routines, you could certainly maintain your own version of it to perform exactly as you specify, or you could use
one of them as a starting point and customize it to suit your needs. You'll probably want to use a different name for
your own replacement routines so that they won't be overwritten during a service pack upgrade.

The SQL Server online documentation explains the specifics of each system routine, so I won't go into detail on each
and every routine here. Since there are many more system stored procedures than system functions, the following
sections discuss the broad categories for grouping stored procedures: General, Catalog, SQL Server Agent,
Replication, and Extended.

General System Procedures

System stored procedures aid in the administration of your system, and they sometimes modify the system tables.
You shouldn't configure the system to allow direct modification of the system tables because a mistake can render
your database useless. That's why direct modification of system tables is prohibited by default. If modification is
necessary, you can use a system stored procedure that is known to do the job correctly.

Below are some of the SQL Server system stored procedures, divided into four categories: basic, security, cursor,
and distributed query procedures. Each procedure's name gives you a clue about its function.

Some basic system procedures:

sp_helptext
sp_helptrigger
sp_attach_db
sp_lock
sp_help
sp_helpconstraint
sp_recompile
This document is created with the unregistered version of CHM2PDF Pilot

sp_helpindex
sp_who

Some of the security stored procedures:

sp_addapprole
sp_defaultlanguage
sp_helplogins
sp_helpntgroup
sp_addlogin
sp_addrolemember
sp_changedbowner
sp_grantdbaccess
sp_changeobjectowner
sp_grantlogin

The following cursor procedures are used to find information about existing cursors:
sp_describe_cursor
sp_describe_cursor_columns
sp_describe_cursor_tables
sp_cursor_list

Finally, here are some of the stored procedures available for managing distributed queries. Distributed queries access
data from multiple heterogeneous data sources. You can store this data in the same computer or different computers,
and you can access it by using OLE DB.
sp_addlinkedserver
sp_addlinkedsrvlogin
sp_serveroption

Catalog Stored Procedures

Applications and development tools commonly need access to information about table names, column types,
datatypes, constraints, privileges, and configuration options. All this information is stored in the system tables, which
are also known as the system catalogs. But system tables might require changes between releases to support new
features, so directly accessing the system tables could result in your application breaking from a new SQL Server
release.

For this reason, SQL Server provides catalog stored procedures, a series of stored procedures that extract the
information from the system tables, providing an abstraction layer that insulates your application. If the system tables
are changed, the stored procedures that extract and provide the information will also be changed to ensure that they
operate consistently (from an external perspective) from one release to another. Many of these procedures also map
nearly identically to ODBC calls. The SQL Server ODBC driver calls these procedures in response to those function
calls. While it's fine to directly query the system catalogs for ad hoc use, if you're deploying an application that needs
to get information from the system tables, you should use these catalog stored procedures. Here's a partial list:
sp_column_privileges
sp_columns
sp_stored_procedures
sp_databases
sp_table_privileges
sp_tables
This document is created with the unregistered version of CHM2PDF Pilot

SQL Server Agent Stored Procedures

SQL Server Enterprise Manager uses SQL Server Agent stored procedures to set up alerts and schedule tasks for
execution. If your application needs to carry out tasks such as these, you can call the following procedures directly.
They must be called from or qualified by the msdb database. Here are some of the SQL Server Agent stored
procedures:
sp_add_alert
sp_add_job
sp_add_jobstep
sp_helptask
sp_delete_alert
sp_start_job
sp_stop_job
sp_help_alert

Web Assistant Procedures

SQL Server provides a number of procedures for creating and managing Web pages. These procedures are typically
called from within the Web Assistant Wizard, but they're documented and you can call them directly. You can
consider them a subcategory of the SQL Server Agent procedures because SQL Server Agent handles much of the
automatic updating of Web pages. Here are the Web assistant procedures:
sp_dropwebtask
sp_enumcodepages
sp_makewebtask
sp_runwebtask

Replication Stored Procedures

You use replication stored procedures to set up and manage publication and subscription tasks. SQL Server
Enterprise Manager typically provides a front end to these procedures, but you can also call them directly. SQL
Server has many replication stored procedures; frankly, it's hard to manually use replication with these procedures.
You can do it, though, if you're determined. Everything SQL Server Enterprise Manager does ultimately uses these
system stored procedures. I urge you to use SQL Server Enterprise Manager or SQL-DMO if you need to add
customized replication administration to your application.

Extended Stored Procedures

You can use extended stored procedures to create your own external routines in a language such as C and have
SQL Server automatically load and execute those routines just like a regular stored procedure. As with stored
procedures, you can pass parameters to extended stored procedures, and they can return result sets, return status
values, or both. This allows you to extend SQL Server capabilities in powerful ways. Many SQL Server features
introduced in the last few years have been implemented using extended stored procedures. These features include
additions to SQL Server Enterprise Manager, login integration with Windows NT and Windows 2000 domain
security, the ability to send and receive e-mail messages, and the ability to create a Web page based on a query.

Extended stored procedures are DLLs that SQL Server can dynamically load and execute. They aren't separate
processes spawned by SQL Server—they run directly in the address space of SQL Server. The DLLs are created
using the Open Data Services (ODS) API, which SQL Server also uses.

Writing an extended stored procedure sounds harder than it really is, which is probably why these procedures are
This document is created with the unregistered version of CHM2PDF Pilot

somewhat underused. But writing one can be as simple as writing a wrapper around a C function. For example,
consider the formatting capabilities in SQL Server's PRINT statement, which are limited and don't allow parameter
substitution. The C language provides the sprintf function, which is powerful for formatting a string buffer and includes
parameter substitution. It's easy to wrap the C sprintf function and create an extended stored procedure that calls it,
resulting in the procedure xp_sprintf. To show you how easy this is, below is the entire source code for the
procedure xp_sprintf. Note that most of this code is setup code, and at the heart is the call to the C run-time
function sprintf:
// XP_SPRINTF
//
// Format and store a series of characters and values into an
// output string using sprintf
//
// Parameters:
// srvproc - the handle to the client connection
//
// Returns:
// XP_NOERROR or XP_ERROR
//
// Side Effects:
//
//
SRVRETCODE xp_sprintf( SRV_PROC * srvproc )
{
int numparams;
int paramtype;
int i;
char string[MAXSTRLEN];
char format[MAXSTRLEN];
char values[MAXARGUMENTS][MAXSTRLEN];
char szBuffer[MAXSTRLEN];

// Get number of parameters

//
numparams=srv_rpcparams(srvproc);

// Check number of parameters

//
if (numparams < 3)
{
// Send error message and return
//
LoadString(hModule, IDS_ERROR_PARAM, szBuffer,
sizeof(szBuffer));
goto ErrorExit;
}

if (paramtype != SRVVARCHAR)
{
// Send error message and return
//
LoadString(hModule, IDS_ERROR_PARAM_TYPE, szBuffer,
sizeof(szBuffer));
goto ErrorExit;
}

if (!srv_paramstatus(srvproc, 1))
{
// Send error message and return
//
LoadString(hModule, IDS_ERROR_PARAM_STATUS, szBuffer,
sizeof(szBuffer));
goto ErrorExit;
}
This document is created with the unregistered version of CHM2PDF Pilot

for (i = 2; i <= numparams; i++)

{
paramtype=srv_paramtype(srvproc, i);

if (paramtype != SRVVARCHAR)
{
// Send error message and return
//
LoadString(hModule, IDS_ERROR_PARAM_TYPE, szBuffer,
sizeof(szBuffer));
goto ErrorExit;
}
}

for (i = 0; i < MAXARGUMENTS; i++)

{
memset(values[i], 0, MAXSTRLEN);

srv_bmove(srv_paramdata(srvproc, i + 3),
values[i],
srv_paramlen(srvproc, i + 3));
}

memset(string, 0, MAXSTRLEN);
srv_bmove(srv_paramdata(srvproc, 2), format,
srv_paramlen(srvproc, 2));
format[srv_paramlen(srvproc, 2)]='\0';

// This is the heart of the function — it simply wraps sprintf

// and passes back the string
sprintf(string, format,
values[0], values[1], values[2], values[3], values[4],
values[5], values[6], values[7], values[8], values[9],
values[10], values[11], values[12], values[13], values[14],
values[15], values[16], values[17], values[18], values[19],
values[20], values[21], values[22], values[23], values[24],
values[25], values[26], values[27], values[28], values[29],
values[30], values[31], values[32], values[33], values[34],
values[35], values[36], values[37], values[38], values[39],
values[40], values[41], values[42], values[43], values[44],
values[45], values[46], values[47], values[48], values[49]);

srv_paramset(srvproc, 1, string, strlen(string));

return XP_NOERROR;

ErrorExit:
srv_sendmsg(srvproc,
SRV_MSG_ERROR,
SPRINTF_ERROR,
SRV_INFO,
(DBTINYINT) 0,
NULL,
0,
0,
szBuffer,
SRV_NULLTERM);

return XP_ERROR;
}

Because extended stored procedures run in the same address space as SQL Server, they can be efficient; however,
This document is created with the unregistered version of CHM2PDF Pilot

a badly behaved extended stored procedure can crash SQL Server, although this is unlikely. A server crash is more
likely to result from someone's maliciousness rather than carelessness. But this is a definite area for concern, and you
should understand the issues that are covered in the rest of this section.

An extended stored procedure runs on the thread that called it. Each calling thread executes using the operating
system's structured exception handling constructs (most notably try-except). When a thread is poorly written and
performs a bad operation, such as trying to reference memory outside its address space, it is terminated. But only that
single connection is terminated, and SQL Server remains unaffected. Any resources held by the thread, such as locks,
are automatically released.

In actual use, extended stored procedures don't introduce significant stability problems into the environment.
Nonetheless, it's certainly possible for an extended stored procedure to twiddle some data structure within SQL
Server (to which it would have access because the procedure is part of SQL Server's address space) that could
disrupt SQL Server's operation or even corrupt data. If you're unlucky, this could happen as the result of a bug in the
extended stored procedure; however, it's more likely that the procedure would cause an access violation and have its
thread terminated with no ill effects.

A procedure could conceivably cause data corruption, but such data structures aren't exposed publicly, so it would
be hard to write a malicious procedure. It is possible, however, and given the propensity of some malicious people to
create viruses, we can't rule this problem out (although no such cases have been documented). The ultimate
responsibility for protecting your data has to rest with your system administrator, who has control over which, if any,
extended stored procedures can be added to the system.
A Word of Warning
Although extended stored procedures can be terrific additions to your applications and in most cases do not
negatively impact your system's behavior, exceptions do exist. You can save yourself hours of grief if you keep the
following points in mind:
 Be sure to include full error checking and exception handling. An unhandled exception will usually bring down
SQL Server and generate a Dr. Watson dump.
 Be sure to stress test your extended procedures thoroughly. Even though they're not running on SQL Server
itself, don't assume that you don't have to be concerned about how SQL Server will behave with hundreds or
thousands of users accessing it.
 Read and follow the guidelines in Microsoft Knowledge Base article Q190987, entitled "Extended Stored
Procedures: What Everyone Should Know," which is included on the companion CD.

Only someone with the sysadmin role can register an extended stored procedure with the system (using
sp_addextendedproc), and only a system administrator can grant others permission to execute the procedure.
Extended stored procedures can be added only to the master database (eliminating their ability to be easily
transferred to other systems via a backup and restore of databases, for example). Administrators should allow use of
only the procedures that have been thoroughly tested and proven to be safe and nondestructive.

Ideally, administrators could also have access to the source code and build environment of the extended stored
procedure to verify that it bears no malicious intent. (Some people say they don't even want their SQL Server
administrator to be able to do this, because that person might not be trustworthy. If that's the case, you have bigger
problems. If you can't trust your system administrator, you'd better get a new one!)

Even without extended stored procedures, an administrator can disrupt a SQL Server environment in many ways.
(Munging the system tables would be a good start.) Of course, you can decide that no one will ever add extended
stored procedures to your system. That's certainly a safe approach, but you give up a powerful capability by taking
this route. (It's kind of like deciding never to ride in a car in order to avoid an accident.)

Even if you prohibit foreign extended stored procedures from your system, don't go overboard with a sweeping rule
that prevents the use of even the procedures provided by Microsoft to implement new features. Could one of these
procedures have a bug that could disrupt SQL Server? Sure, but a bug is no more likely to occur than if the code for
these procedures had simply been statically linked into the SQLSERVER.EXE file rather than implemented as a DLL
This document is created with the unregistered version of CHM2PDF Pilot

and loaded on demand. Of course, Microsoft procedures are thoroughly tested before their release. The chance of a
catastrophic bug occurring is pretty low. The fact that these are extended stored procedures in no way increases the
risk of bugs. It's an engineering decision—and a smart one—that allows Microsoft to add more features to the
product in a way that doesn't require extra change to the core product or additional resource use by environments that
don't call these features.

By convention, most of the extended stored procedures provided with the product begin with xp_. Unlike the sp_
prefix, no special properties are associated with xp_. In fact, several extended stored procedures begin with sp_ (for
example, sp_getbindtoken), which allows them to be called from any database without being fully qualified, so we
could just call it with EXEC sp_getbindtoken instead of EXEC master.dbo.xp_getbindtoken. To ascertain whether
a procedure is a regular stored procedure or an extended stored procedure, you shouldn't rely on the prefix of the
name. Use the function OBJECTPROPERTY. For example, the following should return 1, indicating that
sp_getbindtoken is an extended procedure:
USE master
SELECT OBJECTPROPERTY(OBJECT_ID('sp_getbindtoken'), 'IsExtendedProc')

If you substitute a "real" stored procedure name for sp_getbindtoken, such as sp_help, the function will return a 0.

As with stored procedures, some extended stored procedures that come with the product are not documented for
direct use. These procedures exist to support functionality elsewhere—especially for SQL Server Enterprise
Manager, SQL-DMO, and replication—rather than to provide features directly themselves.

The following extended stored procedures are provided and documented for direct use. First, here are the general
extended stored procedures:
xp_cmdshell
xp_sprintf
xp_sscanf

Here are some of the administration and monitoring extended stored procedures:
xp_logevent
xp_msver
xp_sqlmaint

These are the integrated security-related extended stored procedures:

xp_enumgroups
xp_grantlogin
xp_loginconfig
xp_logininfo
xp_revokelogin

And finally, here are the SQL mail-related extended stored procedures:
xp_deletemail
xp_findnextmsg
xp_readmail
xp_sendmail
xp_startmail
xp_stopmail

SQL Profiler Extended Procedures

This document is created with the unregistered version of CHM2PDF Pilot

SQL Profiler provides a graphical user interface to a set of extended stored procedures for monitoring dozens of
aspects of SQL Server's internal behavior and for creating traces. You can also use extended stored procedures to
create your own applications that monitor SQL Server. In Chapter 17, I'll discuss SQL Profiler and describe many of
the extended procedures available for working with traces.
This document is created with the unregistered version of CHM2PDF Pilot

34

Execute("ANY STRING")
The ability to formulate and then execute a string dynamically in SQL Server is a subtle but powerful capability. Using
this capability, you can avoid additional round-trips to the client application by formulating a new statement to be
executed directly on the server. You can pass a string directly, or you can pass a local variable of type char or
varchar. This capability is especially useful if you need to pass an object name to a procedure or if you want to build
an SQL statement from the result of another SQL statement.

For example, suppose that we had partitioned our database to have multiple tables similar to the authors table. We
could write a procedure like the one shown below to pass the name of the table we want to insert into. The procedure
would then formulate the INSERT statement by concatenating strings, and then it would execute the string it
formulated:
CREATE PROC add_author
@au_id char(11),
@au_lname varchar(20),
@au_fname varchar(20),
@tabname varchar(30) AS

BEGIN
DECLARE @insert_stmt varchar(255)
SELECT @insert_stmt='INSERT ' + @tabname + ' (au_id,
au_lname, au_fname, contract) VALUES (''' + @au_id +
''',''' + @au_lname + ''',''' + @au_fname + ''', 1)'
-- PRINT @insert_stmt
EXECUTE (@insert_stmt)
END

EXEC add_author '999-99-1234', 'Pike', 'Neil', 'authors'

Note that the procedure uses lots of single quotes. I didn't want to use double quotes because the option
QUOTED_IDENTIFIER is ON by default in SQL Query Analyzer, and that means you can use double quotes only
to indicate object names. Two single quotes together, inside of a string, will be interpreted as a single quote. It's the
way to "quote the quote." In addition, you might notice that I included a simple debugging technique. I commented a
statement that will actually show me the value I built in the string @insert_stmt. If SQL Server complains of syntax
errors when the procedure is executed and I just can't figure out where the error is, I can ALTER the procedure,
uncomment the PRINT statement, and put the comment marks in front of the EXECUTE statement.

If you make use of this dynamic EXECUTE inside a routine, there are a few additional issues that you need to be
aware of:
 The plan for the SQL statement constructed from the string is not saved as part of the routine's plan. If the
values of any of the variables or parameters used to build the SQL string change, the SQL statement will need
to be recompiled. This means you won't always get the full benefit of having the plans for routines
precompiled.
 Although SQL Server can use variables from outside the EXEC string to construct the SQL statement, any
variables assigned a value in the constructed string will not be available outside the EXEC.
 A USE statement issued to change your database context is in effect only for the EXEC string's execution.
The example below illustrates this behavior:
SET NOCOUNT ON
USE pubs
PRINT db_name()
This document is created with the unregistered version of CHM2PDF Pilot

EXEC ('USE Northwind PRINT db_name() SELECT count(*) FROM products')

PRINT db_name()
GO

RESULTS:
pubs
Northwind

-----------
77

pubs

 Normally, if a user has permission to execute a stored procedure or function, that user will also have implied
permission to access all objects in the routine owned by the routine's owner. This does not apply to objects
accessed inside an EXEC string. Even if a user has permission to execute a procedure, if the user doesn't
have permission to perform all the actions specified in the EXEC string, a run-time error will occur.
 The only operation that you can use to build the string inside the EXEC is the concatenation operator. You
cannot use functions. For example, if you want to select from a table that has the name of the user plus the
day of the month for its name (for example: kalen20), you must build the string to be executed outside of the
EXEC. Here's an example:
DECLARE @sqlstring varchar(100)
SELECT @sqlstring = 'SELECT * FROM ' + LTRIM(user_name()) +
CONVERT(char(2),DATEPART(dd, GETDATE()))
EXEC (@sqlstring)
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
Transact-SQL statements can be grouped together in batches, they can persist in the database, and they can
repeatedly execute as stored procedures or return values within a statement as functions. It is essential that you
understand the differences between these programming constructs and that you understand that their actions are not
mutually exclusive.

Transact-SQL routines can be quite complex, and they can become a significant portion of your application's source
code. However, the hardest part of writing routines is writing the SQL code that they are composed of.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 12

Transactions and Triggers

In this chapter, we'll continue looking at the use of Transact-SQL for creating programmable modules. In particular,
we'll look at modules that automatically execute when some data modification statement is issued. This type of module
is called a trigger. You can define a trigger to execute as an alternative to the requested data modification or to
execute after the modification takes place. To thoroughly understand how triggers behave, you need to have a fuller
understanding of transactions and transaction boundaries. This chapter will go into more detail about transactions and
how they affect changes made by multiple users.
This document is created with the unregistered version of CHM2PDF Pilot

34

Transactions
Like a batch (discussed in Chapter 11), a user-declared transaction typically consists of several SQL commands that
read and update the database. However, a batch is basically a client-side concept; it controls how many statements
are sent to SQL Server for processing at once. A transaction, on the other hand, is a server-side concept. It deals
with how much work SQL Server will do before it considers the changes committed. A multistatement transaction
doesn't make any permanent changes in a database until a COMMIT TRANSACTION statement is issued. In
addition, a multistatement transaction can undo its changes when a ROLLBACK TRANSACTION statement is
issued.

Both batches and transactions can contain multiple commands. In fact, transactions and batches can have a
many-to-many relationship. A single transaction can span several batches (although that's a bad idea from a
performance perspective), and a batch can contain multiple transactions. The following examples illustrate these
possibilities. Note that you can use TRAN as an abbreviation for TRANSACTION.

The first script is one transaction spread across four batches:

BEGIN TRAN
INSERT authors VALUES (etc.)
GO
SELECT * FROM authors
GO
UPDATE publishers SET pub_id = (etc.)
GO
COMMIT TRAN
GO

This second script is one batch that contains two transactions:

BEGIN TRAN
INSERT authors VALUES (etc.)
SELECT * FROM authors
COMMIT TRAN
BEGIN TRAN
UPDATE publishers SET pub_id = (etc.)
INSERT publishers VALUES (etc.)
COMMIT TRAN
GO

Now let's look at a simple transaction in a little more detail. The BEGIN TRAN and COMMIT TRAN statements
cause the commands between them to be performed as a unit:
BEGIN TRAN
INSERT authors VALUES (etc.)
SELECT * FROM authors
UPDATE publishers SET pub_id = (etc.)
COMMIT TRAN
GO

Transaction processing in SQL Server ensures that all commands within a transaction are performed as a unit of
work—even in the presence of a hardware or general system failure. Such transactions are considered to have the
This document is created with the unregistered version of CHM2PDF Pilot

ACID properties (atomicity, consistency, isolation, and durability). (For more information about the ACID properties,
see Chapter 2.)

Explicit and Implicit Transactions

By default, SQL Server treats each statement—whether it's dispatched individually or as part of a batch—as
independent and immediately commits it. Logically, the only kind of statement that can be committed is a statement
that changes something in the database, so talking about committing a SELECT statement is really meaningless. When
we look at transaction isolation levels later in this section, you'll see that it's important in some cases to include
SELECTs inside larger transactions. However, when we talk about transactions, we're generally talking about data
modification statements (INSERT, UPDATE, and DELETE). So any individual data modification statement by itself is
an implicit transaction. No matter how many rows are modified, they will either all be modified or none of them will
be. If a system failure occurs while a million-row table is being updated, when SQL Server recovers, it will roll back
the part of the update that's already been written to disk, and the state of the database will be as if the update never
happened.

If you want a transaction to include multiple statements, you must wrap the group of statements within BEGIN
TRANSACTION and COMMIT TRANSACTION or ROLLBACK TRANSACTION statements. Most of the
examples in this section will deal with transactions containing multiple statements, which are called explicit
transactions.

You can also configure SQL Server to implicitly start a transaction by using SET IMPLICIT_TRANSACTIONS
ON or by turning on the option globally using sp_configure 'user options', 2. More precisely, you take the previous
value for the user options setting and OR it with (decimal) 2, which is the mask for IMPLICIT_TRANSACTIONS.

For example, if the previous value were (decimal) 8, you'd set it to 10 because 8|2 is 10. (The symbol used here is
the vertical bar, not a forward or backward slash.) If bit operations such as 8|2 are somewhat foreign to you, let SQL
Server do the work. You can issue a SELECT 8|2 in SQL Query Analyzer, and the value returned will be 10. (But
be careful not to assume that you just add 2 to whatever is already there—for example, 10|2 is 10, not 12.)

If implicit transactions are enabled, all statements are considered part of a transaction and no work is committed until
and unless an explicit COMMIT TRAN (or synonymously, COMMIT WORK) is issued. This is true even if all the
statements in the batch have executed: you must issue a COMMIT TRANSACTION in a subsequent batch before
any changes are considered permanent. The examples in this book assume that you have not set
IMPLICIT_TRANSACTIONS on and that any multistatement transaction must begin with BEGIN TRAN.

Error Checking in Transactions

One of the most common mistakes that developers make with SQL Server is to assume that any error within a
transaction will cause the transaction to automatically roll back. This is an understandable misconception because
we're always talking about transactions being atomic. However, when we talk about a failure causing a transaction to
be rolled back, it usually means a system failure. If the whole system stops functioning, SQL Server must be restarted
and the automatic recovery process will roll back any incomplete transactions. However, for many errors encountered
in a transaction, only the current statement will be aborted—the rest of the statements in the transaction can continue
to be processed and even committed. Here's an example:
USE PUBS
BEGIN TRAN
UPDATE authors
SET state = 'FL'
WHERE state = 'KS'

UPDATE jobs
SET min_lvl = min_lvl - 10
This document is created with the unregistered version of CHM2PDF Pilot

COMMIT TRAN

SELECT * FROM authors

WHERE state = 'FL'

The transaction contains two UPDATE statements. The first UPDATE changes the state value in one row to 'FL'.
The second UPDATE attempts to subtract 10 from all the min_lvl values in the jobs table, but there is a CHECK
constraint on the min_lvl column that specifies that min_lvl must be greater than or equal to 10. Most of the values in
the jobs table are greater than 20, so subtracting 10 would not violate the constraint. However, there is one row with
a value of 10, and subtracting 10 would cause a constraint violation. Because of that one conflicting row, the entire
statement is aborted and no rows in jobs are updated, but the transaction is not rolled back. The SELECT statement
after the COMMIT shows that we still have one author with a value for state of 'FL'.

Your multistatement transactions should check for errors by selecting the value of @@ERROR after each statement.
If a nonfatal error is encountered and you don't take action on it, processing moves to the next statement. Only fatal
errors cause the batch to be automatically aborted and the transaction rolled back. Here's what the preceding batch
would look like with the appropriate error checking:
USE PUBS
BEGIN TRAN
UPDATE authors
SET state = 'FL'
WHERE state = 'KS'

IF @@ERROR <> 0 BEGIN

ROLLBACK TRAN
GOTO ON_ERROR
END
UPDATE jobs
SET min_lvl = min_lvl - 10
IF @@ERROR <> 0 BEGIN
ROLLBACK TRAN
GOTO ON_ERROR
END

COMMIT TRAN
ON_ERROR:

SELECT * FROM authors

WHERE state = 'FL'

In this batch, we're checking for errors after each of the data modification statements. Just rolling back the transaction
if an error is found is not sufficient, however, because in most cases a ROLLBACK TRAN does not change the flow
of control in the execution of your statements. The ROLLBACK TRAN would undo any completed modifications in
the current transaction and make the appropriate entries in the transaction log. However, program flow would
continue to the next line. In this case, with no GOTO, if the first UPDATE had an error we would still continue to the
second UPDATE; if the second UPDATE had an error, we would still continue and try to execute the COMMIT
TRAN. Since we would have already executed a ROLLBACK TRAN, there would no longer be an open
transaction to commit and we would get an error for the COMMIT TRAN statement.
TIP
A query that finds no rows meeting the criteria of the WHERE clause is not an error. Nor is a searched UPDATE
statement that affects no rows. @@ERROR returns a 0 (meaning no error) in either case. If you want to check for
"no rows affected," use @@ROWCOUNT, not @@ERROR.

Syntax errors always cause the entire batch to be aborted before execution even begins, and references to objects
that don't exist (for example, a SELECT from a table that doesn't exist) cause a batch to stop execution at that point.
This document is created with the unregistered version of CHM2PDF Pilot

However, just because a batch has been aborted doesn't mean that the transaction has been rolled back. You've seen
that transactions can span batches. If you receive an error and you're not sure whether your transaction has been
rolled back, you can check the status of the transaction using the system function @@TRANCOUNT. If
@@TRANCOUNT is greater than 0, there is an active transaction and you can execute a ROLLBACK TRAN. I'll
discuss @@TRANCOUNT in more detail later in this section. Typically, you'll work out syntax errors and correct
references to objects before you put an application into production, so these won't be much of an issue.

A syntax error on a batch dynamically built and executed using EXECUTE('string') can never be caught until
execution. You can think of this EXECUTE('string') construct as a way of nesting batches. If the string consists of
multiple statements, a syntax or object reference error will abort the inner batch only. Processing proceeds to the next
statement in the outer batch. For example, consider the following script:
DECLARE @tablename sysname
SET @tablename = 'authours'
EXECUTE ('USE pubs SELECT * FROM ' + @tablename +
' PRINT ''Inner Batch Done'' ')
PRINT 'Outer Batch Done'

This batch won't have an error detected at parse time, so SQL Server will attempt to execute it. The DECLARE and
the SET will be executed successfully, but the nested batch inside the EXECUTE will have a problem. There's an
error in the SELECT because the table name authours doesn't exist in the pubs database. The inner batch will
generate an error during its name resolution phase, so the third statement in the inner batch (PRINT) will not be
executed. However, the outer batch will continue and the message "Outer Batch Done" will be printed.

Errors that occur in a production environment are typically resource errors. You probably won't encounter these
much, especially if you've done sufficient testing to make sure that your configuration settings and environment are
appropriate. Out-of-resource errors occur when a database file fills up, when there isn't enough memory to run a
procedure, when a database is off line and unavailable, and so on. For these types of fatal errors, conditional action
based on @@ERROR is moot because the batch will automatically be aborted.

The following are among the most common errors that you need to be concerned about:
 Lack of permissions on an object
 Constraint violations
 Duplicates encountered while trying to update or insert a row
 Deadlocks with another user
 NOT NULL violations
 Illegal values for the current datatype

Here's an example that creates three tables, two of which have a primary/foreign key relationship. Table b has a
foreign key reference to table a; any INSERT into b should fail if the value inserted doesn't already exist in table a. A
procedure then tries to insert into two of the tables. The procedure is written to perform error checking, with a branch
to perform a rollback if any error is encountered. Since no inserts have been done into table a, any insert into b should
fail:
CREATE TABLE a (
a char(1) primary key)

CREATE TABLE b (
b char(1) references a)

CREATE TABLE c (
c char(1))
GO

CREATE PROC test as

BEGIN TRANSACTION
INSERT c VALUES ('X')
This document is created with the unregistered version of CHM2PDF Pilot

IF (@@ERROR <> 0) GOTO on_error

INSERT b VALUES ('X') -- Fails reference
IF (@@ERROR <> 0) GOTO on_error
COMMIT TRANSACTION
RETURN(0)

on_error:
ROLLBACK TRANSACTION
RETURN(1)

This simple procedure illustrates the power of Transact-SQL. The system function @@ERROR can return a value
for the connection after each statement. A value of 0 for @@ERROR means that no error occurred. Given the data
the user provided, when the procedure test is executed, the INSERT statement on table b will fail with a foreign key
violation. The error message for that type of failure is error 547, with text such as this:
INSERT statement conflicted with COLUMN FOREIGN KEY constraint
'FK__b__b__723BFC65'. The conflict occurred in database 'pubs',
table 'a', column 'a'

Consequently, @@ERROR would be set to 547 following that INSERT statement. Therefore, the statement IF
(@@ERROR <> 0) evaluates as TRUE, and execution follows the GOTO to the on_error: label. Here, the
transaction is rolled back. The procedure terminates with a return code of 1 because the RETURN(1) statement was
used. Had the branch to on_error: not been followed (by virtue of @@ERROR not being 0), the procedure would
have continued line-by-line execution. It would have reached COMMIT TRANSACTION, and then it would have
returned value 0 and never made it all the way to the on_error: section.

As with most programming languages, with Transact-SQL you can return a status code from a procedure to indicate
success or failure (or other possible outcomes); those return status codes are checked from the calling routines.
However, you have to remember to check the status after executing the procedures. Merely using EXEC test won't
directly provide information about whether the procedure performed as expected. A better method is to use a local
variable to examine the return code from that procedure:
DECLARE @retcode int
EXEC @retcode=test

Following execution of the test procedure, the local variable @retcode has the value 0 (if no errors occurred in the
procedure) or 1 (if execution branched to the on_error: section).

The SET XACT_ABORT option was added to SQL Server to force any error—not just a fatal error—to terminate
the batch. Here's another way to ensure that nothing is committed if any error is encountered:
CREATE PROC test AS
SET XACT_ABORT ON
BEGIN TRANSACTION
INSERT c VALUES ('X')
INSERT b VALUES ('X') -- Fails reference
COMMIT TRANSACTION
GO

EXEC test
GO

SELECT * FROM c

Here's the output:

This document is created with the unregistered version of CHM2PDF Pilot

(0 rows affected)

Note that the name of the XACT_ABORT option is a bit of a misnomer because the current batch, not simply the
transaction, is immediately aborted if an error occurs, just as it is when a fatal resource error is encountered. This has
consequences that might not be immediately apparent. For example, if you issued two transactions within one batch,
the second transaction would never be executed because the batch would be aborted before the second transaction
got a chance to execute. More subtly, suppose we want to use good programming practice and check the return
status of the procedure above. (Note that even though it doesn't explicitly do a RETURN, every procedure has a
return status by default, with 0 indicating SUCCESS.)

We could write a batch like this:

DECLARE @retcode int
EXEC @retcode=test
SELECT @retcode

Yet there's a subtle but important problem here. If the procedure has an error, the SELECT @RETCODE statement
will never be executed: the entire batch will be aborted by virtue of the SET XACT_ABORT statement. This is why I
recommend checking @@ERROR instead of using SET XACT_ABORT. Checking @@ERROR after each
statement is a bit more tedious, but it gives you finer control of execution in your procedures.

Unfortunately, error handling in SQL Server 2000 can be somewhat messy and inconsistent. For example, there's no
way to install a routine that means "Do this on any error" (other than SET XACT_ABORT, which aborts but doesn't
let you specify the actions to be performed). Instead, you must use something similar to the preceding examples that
check @@ERROR and then do a GOTO.

In addition, there's currently no easy way to determine in advance which errors might be considered fatal so that the
batch can be aborted and which errors are nonfatal so that the next statement can be executed. In most cases, an
error with a severity level of 16 or higher is fatal and the batch will be aborted. Syntax that refers to nonexistent
functions are level-15 errors, yet the batch is still aborted. Although you can use @@ERROR to return the specific
error number, no function such as @@SEVERITY is available to indicate the error's severity level. Instead, you must
subsequently select from the sysmessages table to see the severity level of the last error. To further complicate
matters, some level-16 errors aren't fatal, including the constraint violation error encountered in the first
error-checking example. Table 12-1 lists the most common nonfatal level-16 errors. Note the %.*s placeholders in
the error messages: SQL Server substitutes the actual names being referenced when it issues the error message.

Admittedly, the rules are hardly consistent, and this area is ripe for some attention in future releases. As you write
your procedures, keep in mind that they could be automatically aborted due to an unexpected fatal error, or you
could cause them to abort with a nonfatal error.

Table 12-1. Common nonfatal level-16 errors.

Level-16 Error Error Message
Attempt to insert the value NULL into column '%.*s',
515
table '%.*s'; column doesn't allow nulls. %s fails.
Attempt to insert explicit value for identity column in
544
table '%.*s' when IDENTITY_INSERT is set to OFF.
Attempt to execute %s statement conflicted with %s %s
547 constraint '%.*s'. The conflict occurred in database '%.*s'
, table '%.*s'%s%.*s%s.
This document is created with the unregistered version of CHM2PDF Pilot

Attempt to insert or update failed because the target view

either specifies WITH CHECK OPTION or spans a
550 view that specifies WITH CHECK OPTION, and one or
more rows resulting from the operation didn't qualify
under the CHECK OPTION constraint.

Transaction Isolation Levels

The isolation level at which your transaction runs determines your application's sensitivity to changes made by others;
consequently, it also determines how long your transaction needs to hold locks to potentially protect against changes
made by others. SQL Server 2000 offers four isolation-levels:
 READ UNCOMMITTED (dirty read)
 READ COMMITTED (default—READ COMMITTED is equivalent to the term CURSOR STABILITY,
which is used by several other products such as IBM DB/2)
 REPEATABLE READ
 SERIALIZABLE

Your transactions will behave differently depending on which isolation level is set. The saying "Not to decide is to
decide" applies here, because every transaction has an isolation level whether you've specified it or not. You should
understand the levels and choose the one that best fits your needs.
NOTE
The syntactical options listed above correspond to the SQL standard isolation levels, which I discussed in detail in
Chapter 3. In this section, we'll look at information that wasn't covered earlier. See Chapter 3 if you need a refresher.

When you use the READ UNCOMMITTED option, keep in mind that you should deal with inconsistencies that
might result from dirty reads. Because share locks aren't issued and exclusive locks of other connections aren't
honored (that is, the data pages can be read even though they're supposedly locked), it's possible to get some
spurious errors during execution when you use READ UNCOMMITTED. A moment later, reexecuting the same
command is likely to work without error.

Nothing illustrates the differences and effects of isolation levels like seeing them for yourself with some simple
examples. To run the following examples, you'll need to establish two connections to SQL Server. Remember that
each connection is logically a different user, even if both connections use the same login ID. Locks held by one
connection affect the other connection even if both are logged in as the same user. In case you can't run these
examples, the SQL scripts and output are shown here.

EXAMPLE 1

This example shows a dirty read. To test these examples yourself, you have to run the commands for connection 1 in
one SQL Query Analyzer window and the commands for connection 2 in a second window. The commands for each
connection will be listed side by side, and it is important to execute the batches for each connection at the appropriate
time relative to the batches for the other connection. That is, the batches should be executed in the order of the "batch
number" listed in the leftmost column. Following is the code and the output (in bold):
Batch Connection 1 Connection 2
----- ----------------------------------- -------------------------------
1 USE PUBS
GO

2 USE PUBS
GO

3 -- First verify there's only

-- one author named 'Smith'
SELECT au_lname FROM authors
This document is created with the unregistered version of CHM2PDF Pilot

WHERE au_lname='Smith'
GO
RESULTS
au_lname
--------------------------
Smith

(1 row affected)

SET TRANSACTION ISOLATION LEVEL

READ UNCOMMITTED
GO

4 BEGIN TRAN
UPDATE authors
SET au_lname='Smith'
-- Give other connection chance to
-- read uncommitted data
GO
(23 rows affected)

5 SELECT au_lname FROM authors

WHERE au_lname='Smith'

au_lname
--------------------------
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith
Smith

(23 rows affected)

IF (@@ROWCOUNT > 1)
PRINT 'Just read uncommitted
data !!'

Just read uncommitted data !!

6 -- UNDO the changes

-- to the authors table
ROLLBACK TRAN
GO

7 -- Now check again for authors

-- of name 'Smith'.
This document is created with the unregistered version of CHM2PDF Pilot

-- Find only one now, because

-- other connection did a rollback.
SELECT au_lname FROM authors
WHERE au_lname='Smith'

au_lname
--------------------------
Smith

(1 row affected)

These batches, running simultaneously, illustrate that by setting the isolation level to READ UNCOMMITTED, you
can read data that logically never existed (a dirty read). The update to Smith was never committed by connection 1,
yet the other connection read it. Not only did the connection read the updated data, but it did so immediately and
didn't have to wait for the exclusive lock of the first connection updating the data to be released.

In this example, concurrency is high; however, consistency of the data isn't maintained. By changing the isolation level
back to the default (READ COMMITTED), the same batches would never see more than one row containing Smith.
But with READ COMMITTED, the connection reading the data must wait until the updating transaction is done. This
is the trade-off: higher consistency is achieved (the uncommitted rows aren't seen by others), but concurrency is
reduced.

EXAMPLE 2

This example illustrates the semantic differences between selecting data under the default isolation level of READ
COMMITTED (cursor stability) and under the isolation level of REPEATABLE READ.
Batch Connection 1 Connection 2
----- -------------------- -----------------------------------
1 USE PUBS
GO

2 USE PUBS
GO

3 UPDATE authors SET au_lname='Doe'

-- Make sure no Smith or Jones
(23 rows affected)

-- Change the transaction isolation

-- level and then verify there are
-- no authors named 'Smith'
SET TRANSACTION ISOLATION LEVEL
READ COMMITTED
GO
BEGIN TRAN
SELECT au_lname FROM authors
WHERE au_lname='Smith'
GO
(0 rows affected)

4 UPDATE authors
SET au_lname='Smith'
GO
(23 rows affected)

5 SELECT au_lname FROM authors

WHERE au_lname='Smith'
GO

au_lname
This document is created with the unregistered version of CHM2PDF Pilot

------------------------
Smith
Smith
Smith

Smith
Smith

(23 rows affected)

COMMIT TRAN
GO

-- Now do the same thing,

-- but with REPEATABLE READ
isolation

SET TRANSACTION ISOLATION LEVEL

REPEATABLE READ
GO

BEGIN TRAN
SELECT au_lname FROM authors
WHERE au_lname='Smith'
GO
au_lname
------------------------
Smith
Smith
Smith

Smith
Smith

(23 rows affected)

6 UPDATE authors SET au_lname='Jones'

GO
(query will block)

7 SELECT au_lname FROM authors

WHERE au_lname='Smith'
GO
au_lname
------------------------
Smith
Smith
Smith

Smith
Smith

(23 rows affected)

COMMIT TRAN
GO

8 -- Now notice that query

-- has completed
(23 rows affected)

9 -- Now notice that Jones updates

-- have been done
SELECT au_lname FROM authors
GO
This document is created with the unregistered version of CHM2PDF Pilot

au_lname
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones
Jones

(23 rows affected)

As you can see, when the isolation level was set to READ COMMITTED, selecting the same data twice within a
transaction yielded totally different results: first no Smiths were found, and then 23 of them appeared. If this were a
banking transaction, these results would be unacceptable. For applications that are less sensitive to minor changes, or
when business functions guard against such inconsistencies in other ways, this behavior might be acceptable.

After changing the isolation level to REPEATABLE READ, we got the same result (no Joneses) with both SELECT
statements. Immediately after the transaction was committed, the other connection updated all the names to Jones. So
when we again selected for Jones (immediately after the transaction was committed), the update took place—but at
some cost. In the first case, the other connection was able to do the update as soon as the first SELECT was
processed. It didn't have to wait for the second query and subsequent transaction to complete. In the second case,
that second connection had to wait until the first connection completed the transaction in its entirety. In this example,
concurrency was significantly reduced but consistency was perfect.

EXAMPLE 3

This example illustrates the semantic differences between selecting data under the isolation level of REPEATABLE
READ and under the isolation level of SERIALIZABLE.
Batch Connection 1 Connection 2
----- -------------------------------------- ----------------------------------
1 USE PUBS
GO

2 USE PUBS
GO

3 SET TRANSACTION ISOLATION LEVEL

REPEATABLE READ
GO

BEGIN TRAN
SELECT title FROM titles
WHERE title_id LIKE 'BU%'
This document is created with the unregistered version of CHM2PDF Pilot

GO
title
-----------------------------------
The Busy Executive's Database Guide
Cooking with Computers:
Surreptitious Balance Sheets
You Can Combat Computer Stress!
Straight Talk About Computers

4 INSERT titles VALUES

('BU2000', 'Inside SQL Server 2000',
'popular_comp', '0877', 59.95, 5000,
10, 0, null, 'Sep 10, 2000')
GO
(1 row(s) affected)

5 -- execute the same SELECT again

SELECT title FROM titles
WHERE title_id LIKE 'BU%'
GO
title
-----------------------------------
Inside SQL Server 2000
The Busy Executive's Database Guide
Cooking with Computers:
Surreptitious Balance Sheets
You Can Combat Computer Stress!
Straight Talk About Computers

COMMIT TRAN

-- Now do the same thing,

-- but with SERIALIZABLE isolation

SET TRANSACTION ISOLATION LEVEL

SERIALIZABLE
GO

BEGIN TRAN
SELECT title FROM titles
WHERE title_id LIKE 'BU%'
GO

title
------------------------------------
Inside SQL Server 2000
The Busy Executive's Database Guide
Cooking with Computers:
Surreptitious Balance Sheets
You Can Combat Computer Stress!
Straight Talk About Computers

6 INSERT titles VALUES ('BU3000',

'Itzik and His Black Belt SQL Tricks',
'popular_comp', '0877', 39.95,
10000, 12, 15000, null,
'Sep 15 2000')
GO
(query will block)

7 SELECT title FROM titles

WHERE title_id LIKE 'BU%'
GO

title
This document is created with the unregistered version of CHM2PDF Pilot

-----------------------------------
Inside SQL Server 2000
The Busy Executive's Database Guide
Cooking with Computers:
Surreptitious Balance Sheets
You Can Combat Computer Stress!
Straight Talk About Computers

COMMIT TRAN
GO

8 -- Now notice that query

-- has completed
(1 row affected)

9 -- Now notice that the second

-- new title has been inserted
-- after the COMMIT TRAN
SELECT title FROM titles
WHERE title_id LIKE 'BU%'
GO

title
-----------------------------------
Inside SQL Server 2000
Itzik and His Black Belt SQL Tricks
The Busy Executive's Database Guide
Cooking with Computers:
Surreptitious Balance Sheets
You Can Combat Computer Stress!
Straight Talk About Computers

As you can see, when the isolation level was set to REPEATABLE READ, phantoms were possible. Another
connection could add new data so that reissuing the same SELECT within a transaction would return more rows the
second time. In this case, Inside SQL Server 2000 just mysteriously appeared; it's the phantom row.

After changing the isolation level to SERIALIZABLE, we got the same result (five titles) with both SELECT
statements in the second connection. Immediately after the transaction committed, the first connection could complete
the insert of a new book with the title Itzik and His Black Belt SQL Tricks. So when we again selected all the titles
(immediately after the transaction was committed), the INSERT had taken place—but again, at some cost. In the first
case, the first connection was able to do the INSERT as soon as the second connection's first SELECT was
processed. It didn't have to wait for the second SELECT and for the transaction to commit. In the second case, that
first connection had to wait until the second connection completed the transaction in its entirety. Just as in Example 2,
concurrency was significantly reduced but consistency was perfect.

Another way to look at the difference between these isolation levels is to see what modification operations are
possible once you've selected data within a transaction. In ISOLATION LEVEL REPEATABLE READ, the actual
rows read can't be modified or deleted. However, new rows can be added to the range of data specified in the
WHERE clause of the SELECT statement. All data read is locked so that no modifications can occur. However,
since INSERT is adding new data, the locks on the existing data do not interfere. In ISOLATION LEVEL
SERIALIZABLE, once you have selected data within a transaction, no modifications are possible to that data, or to
the covered range of data, by any other connection. SERIALIZABLE locks ranges of data, including data that didn't
actually exist, so that INSERTs are not allowed.

Other Characteristics of Transactions

In addition to isolation levels, it's important to understand the following characteristics of transactions.
This document is created with the unregistered version of CHM2PDF Pilot

First, a single UPDATE, DELETE, or INSERT/SELECT statement that affects multiple rows is always an atomic
operation; it must complete without error, or else it will be automatically rolled back. For example, if you performed a
single UPDATE statement that updated all rows but one row failed a constraint, the operation would be terminated
with no rows updated. Because there's no way to do error checking for each row within such a statement, any error
rolls back the statement. However, the batch isn't aborted. Execution proceeds to the next available statement in the
batch. This will occur even in INSERT operations that are based on a SELECT statement (INSERT/SELECT or
SELECT INTO).

Alternatively, a single UPDATE statement that updates hundreds of rows could fail to complete because of a system
failure, possibly unrelated to SQL Server. (For example, the power might go out and your UPS system might not
arrive until next Tuesday.) Because SQL Server considers the single UPDATE to be an atomic unit of work, when
SQL Server comes back up and runs recovery on the database, it will be as if the UPDATE had never happened. If
SQL Server doesn't know for sure that a transaction committed, the transaction will be rolled back. Even if 999,999
rows out of a million had been updated before the failure, we're left with nothing because the transaction is an
all-or-nothing operation.

Like the statements mentioned earlier, modifications made by a trigger are always atomic with the underlying data
modification statement. For example, if an update trigger attempts to update data but fails, the underlying data
modification operation is rolled back. Both operations must succeed or neither will.

It might not be obvious, but Example 2 (the REPEATABLE READ example) shows that a transaction can affect
pure SELECT operations in which no data is modified. The transaction boundaries define the statements between
which a specified isolation level will be assured. Two identical SELECT statements might behave differently if they
were executed while wrapped by BEGIN TRAN/COMMIT TRAN with an isolation level of REPEATABLE READ
than they would behave if they were executed together in the same batch, but not within a transaction.

Nested Transaction Blocks

It's syntactically acceptable for blocks of BEGIN TRANSACTION followed by COMMIT TRAN or
ROLLBACK TRAN to be nested within other such blocks. You can also do this kind of nesting with calls to nested
stored procedures. However, the semantics of such a formulation might not be what you'd expect if you think that the
transactions are truly nested: they aren't. But the behavior is reasonable and predictable. A ROLLBACK TRAN rolls
back all levels of the transaction, not only its inner block. A COMMIT TRAN does nothing to commit the transaction
if the statement isn't part of the outermost block. If the COMMIT TRAN is part of the outermost block, it commits all
levels of the transaction.

So the behavior of a COMMIT or a ROLLBACK isn't too orthogonal when the transaction blocks are nested. Only
the outermost COMMIT is able to commit the transaction, but any ROLLBACK will roll back the entire transaction
at all levels. If this weren't true, a ROLLBACK in an outer transaction wouldn't be able to perform its job because the
data would have already been committed. This behavior allows stored procedures (and triggers) to be executed
automatically and in a predictable way without your needing to check the transaction state. A nested stored procedure
that does a ROLLBACK TRAN will roll back the entire transaction, including work done by the top-level procedure.

In addition, the blocks of BEGIN TRAN and COMMIT TRAN or ROLLBACK TRAN are determined only by
what actually executes, not by what's present in the batch. If conditional branching occurs (via IF statements, for
example) and one of the statements doesn't execute, the statement isn't part of a block.

A common misconception about ROLLBACK TRAN, which I briefly discussed earlier, is that it changes the flow of
control, causing, for example, an immediate return from a stored procedure or a batch. However, flow of control
continues to the next statement, which, of course, could be an explicit RETURN. ROLLBACK affects only the actual
data; it doesn't affect local variables or SET statements. If local variables are changed during a transaction or if SET
statements are issued, those variables and options do not revert to the values they had before the transaction started.
Variables, including table variables, and SET options aren't part of transaction control.
This document is created with the unregistered version of CHM2PDF Pilot

The system function @@TRANCOUNT will return the depth of executed BEGIN TRAN blocks. You can think of
the behavior of COMMIT TRAN and ROLLBACK TRAN in this way: ROLLBACK TRAN performs its job for all
levels of transaction blocks whenever @@TRANCOUNT is 1 or greater. A COMMIT TRAN commits changes
only when @@TRANCOUNT is 1.
NOTE
You'll see several examples of nesting transaction blocks. For illustration, the value of @@TRANCOUNT is shown
in the comments. If this discussion isn't totally clear to you, you should work through these examples by hand to
understand why the value of @@TRANCOUNT is what it is, why it changes, and why the behavior of COMMIT
TRAN or ROLLBACK TRAN acts in a way that depends on the @@TRANCOUNT value.

Executing a BEGIN TRAN statement always increments the value of @@TRANCOUNT. If no transaction is active,
@@TRANCOUNT is 0. Executing a COMMIT TRAN decrements the value of @@TRANCOUNT. Executing a
ROLLBACK TRAN rolls back the entire transaction and sets @@TRANCOUNT to 0. Executing either a
COMMIT TRAN or a ROLLBACK TRAN when there's no open transaction (@@trancount is 0) results in error
3902 or 3903, which states that the COMMIT or ROLLBACK request has no corresponding BEGIN
TRANSACTION. In this case, @@trancount is not decremented, so it can never go below 0.

Other errors, not of your making, can be fatal as well—for example, running out of memory, filling up the transaction
log (if you don't have it set to autogrow), or terminating because of a deadlock. Such errors cause an open
transaction to automatically roll back. If that occurs, @@trancount will return 0, signaling that no open transaction
exists.

The following incidents cause fatal errors:

 The system runs out of resources.
 The log runs out of space.
 Deadlock conditions exist.
 Protection exceptions occur (that is, sufficient permissions are unavailable on an object).
 A stored procedure can't run (for example, it's not present or you don't have privileges).
 The maximum nesting level of stored procedure executions has been reached.

You can't plan precisely for all of these situations, and even if you could, a new release could likely add another one.
In a production environment, fatal errors should be few and far between. But as is true in nearly all programming, it's
up to you to decide whether to try to plan for every conceivable problem and then deal with each one specifically or
whether to accept the system default behavior. In SQL Server, the default behavior would typically be to raise an
error for the condition encountered and then, when the COMMIT TRAN or ROLLBACK TRAN is executed, raise
another error that says it has no corresponding BEGIN TRANSACTION.

One possibility is to write some retry logic in your client application to deal with a possible deadlock condition. For
the other error conditions, you might decide to go with the default error behavior. If your applications are deployed
with proper system management, such errors should be nonexistent or rare. You could also easily check
@@TRANCOUNT before executing a ROLLBACK TRAN or COMMIT TRAN to ensure that a transaction to
operate on is open.

The nesting of transaction blocks provides a good reason to not name the transaction in a ROLLBACK statement.
If, in a rollback, any transaction other than the top-level transaction is named, error 6401 will result:

Cannot roll back XXX. No transaction or savepoint of that name was found.

It's fine to name the transaction in the BEGIN TRAN block, however. A COMMIT can also be named, and it won't
prompt an error if it's not paired in the top-level branch because it's basically a NO-OP in that case anyway (except
that it decrements the value returned by @@trancount).

One reason for naming your BEGIN TRAN statements is to allow your COMMIT and ROLLBACK statements to
This document is created with the unregistered version of CHM2PDF Pilot

be self-documenting. ROLLBACK TRAN doesn't require a transaction name, but you can include one as a way of
emphasizing that you are rolling back the entire transaction.

The following batch will result in an error. The statement ROLLBACK TRAN B will fail with error 6401. The
subsequent ROLLBACK TRAN A will succeed because it's the top-level transaction block. Remember, though, that
we'd get the same behavior if we simply used ROLLBACK TRAN instead of ROLLBACK TRAN A.
-- To start with, verify @@TRANCOUNT is 0
SELECT @@TRANCOUNT
BEGIN TRAN A
-- Verify @@TRANCOUNT is 1
SELECT @@TRANCOUNT
-- Assume some real work happens here
BEGIN TRAN B
-- Verify @@TRANCOUNT is 2
SELECT @@TRANCOUNT
-- Assume some real work happens here
ROLLBACK TRAN B
-- @@TRANCOUNT is still 2 because the previous ROLLBACK
-- failed due to error 6401
SELECT @@TRANCOUNT -- Assume some real work happens here
ROLLBACK TRAN A
-- This ROLLBACK succeeds, so @@TRANCOUNT is back to 0
SELECT @@TRANCOUNT

The following example is arguably an improvement over the previous attempt. The first ROLLBACK will execute
and the second ROLLBACK will fail with error 3903:
Server: Msg 3903, Level 16, State 1
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION.

The first ROLLBACK did its job, and there is no open transaction:
-- To start with, verify @@TRANCOUNT is 0
SELECT @@TRANCOUNT
BEGIN TRAN A
-- Verify @@TRANCOUNT is 1
SELECT @@TRANCOUNT
-- Assume some real work happens here
BEGIN TRAN B
-- Verify @@TRANCOUNT is 2
SELECT @@TRANCOUNT
-- Assume some real work happens here
ROLLBACK TRAN -- Notice the tran is unnamed but works
-- That ROLLBACK terminates transaction. @@TRANCOUNT is now 0.
SELECT @@TRANCOUNT
-- The following ROLLBACK will fail because there is no open
-- transaction (that is, @@TRANCOUNT is 0)
ROLLBACK TRAN
-- @@TRANCOUNT does not go negative. It remains at 0.
SELECT @@TRANCOUNT

If you syntactically nest transactions, be careful not to nest multiple ROLLBACK statements in a way that would
allow more than one to execute. To illustrate that COMMIT behaves differently than ROLLBACK, note that the
following example will run without error. However, the statement COMMIT TRAN B doesn't commit any changes. It
does have the effect of decrementing @@trancount, however.
-- To start with, verify @@TRANCOUNT is 0
SELECT @@TRANCOUNT
This document is created with the unregistered version of CHM2PDF Pilot

BEGIN TRAN A
-- Verify @@TRANCOUNT is 1
SELECT @@TRANCOUNT
-- Assume some real work happens here
BEGIN TRAN B
-- Verify @@TRANCOUNT is 2
SELECT @@TRANCOUNT
-- Assume some real work happens here
COMMIT TRAN B
-- The COMMIT didn't COMMIT anything, but does decrement
-- @@TRANCOUNT
-- Verify @@TRANCOUNT is back down to 1
SELECT @@TRANCOUNT
-- Assume some real work happens here
COMMIT TRAN A
-- closes the transaction.
-- Since there's no open transaction, @@TRANCOUNT is again 0.
SELECT @@TRANCOUNT

In summary, nesting transaction blocks is perfectly valid, but you must understand the semantics. If you understand
when @@trancount is incremented, decremented, and set to 0 and you know the simple rules for COMMIT TRAN
and ROLLBACK TRAN, you can pretty easily produce the effect you want.

Savepoints
Often, users will nest transaction blocks only to find that the resulting behavior isn't what they want. What they really
want is for a savepoint to occur in a transaction. A savepoint provides a point up to which a transaction can be
undone—it might have been more accurately named a "rollback point." A savepoint doesn't "save" or commit any
changes to the database—only a COMMIT statement can do that.

SQL Server allows you to use savepoints via the SAVE TRAN statement, which doesn't affect the
@@TRANCOUNT value. A rollback to a savepoint (not a transaction) doesn't affect the value returned by
@@TRANCOUNT, either. However, the rollback must explicitly name the savepoint: using ROLLBACK TRAN
without a specific name will always roll back the entire transaction.

In the first nested transaction block example shown previously, a rollback to a transaction name failed with error
6401 because the name wasn't the top-level transaction. Had the name been a savepoint instead of a transaction, no
error would have resulted, as this example shows:
-- To start with, verify @@TRANCOUNT is 0
SELECT @@TRANCOUNT
BEGIN TRAN A
-- Verify @@TRANCOUNT is 1
SELECT @@TRANCOUNT
-- Assume some real work happens here
SAVE TRAN B
-- Verify @@TRANCOUNT is still 1. A savepoint does not affect it.
SELECT @@TRANCOUNT
-- Assume some real work happens here
ROLLBACK TRAN B
-- @@TRANCOUNT is still 1 because the previous ROLLBACK
-- affects just the savepoint, not the transaction
SELECT @@TRANCOUNT
-- Assume some real work happens here
ROLLBACK TRAN A
-- This ROLLBACK succeeds, so @@TRANCOUNT is back to 0
SELECT @@TRANCOUNT
This document is created with the unregistered version of CHM2PDF Pilot

34

Triggers
A trigger is a special type of stored procedure that is fired on an event-driven basis rather than by a direct call. Here
are some common uses for triggers:
 To maintain data integrity rules that extend beyond simple referential integrity
 To implement a referential action, such as cascading deletes
 To maintain an audit record of changes
 To invoke an external action, such as beginning a reorder process if inventory falls below a certain level or
sending e-mail or a pager notification to someone who needs to perform an action because of data changes

You can set up a trigger to fire when a data modification statement is issued—that is, an INSERT, UPDATE, or
DELETE statement. SQL Server 2000 provides two types of triggers: after triggers and instead-of triggers.

After Triggers
After triggers are the default type of trigger in SQL Server 2000, so you don't have to use the word AFTER in the
CREATE TRIGGER statement. In addition, if any documentation discusses "triggers" without specifying after or
instead-of triggers, you can assume that the discussion is referring to after triggers.

You can define multiple after triggers on a table for each event, and each trigger can invoke many stored procedures
as well as carry out many different actions based on the values of a given column of data. However, you have only a
minimum amount of control over the order in which triggers are fired on the same table.

For example, if you have four INSERT triggers on the inventory table, you can determine which trigger will be the
first and which will be the last, but the other two triggers will fire in an unpredictable order. Controlling trigger order is
a feature that is needed for some of SQL Server's own internal functionality. For example, when you set up merge
replication, you can designate a trigger as the first trigger to fire. In general, I recommend that you don't create
multiple triggers with any expectation of a certain execution order. If it's crucial that certain steps happen in a
prescribed order, you should reconsider whether you really need to split the steps into multiple triggers.

You can create a single after trigger to execute for any or all the INSERT, UPDATE, and DELETE actions that
modify data. Currently, SQL Server offers no trigger on a SELECT statement because SELECT does not modify
data. In addition, after triggers can exist only on base tables, not on views. Of course, data modified through a view
does cause an after trigger on the underlying base table to fire. As you'll see, you can define instead-of triggers on
views.

An after trigger is executed once for each UPDATE, INSERT, or DELETE statement, regardless of the number of
rows it affects. Although some people assume that a trigger is executed once per row or once per transaction, these
assumptions are incorrect, strictly speaking. Let's see a simple example of this. The script below creates and
populates a small table with three rows. It then builds a trigger on the table to be fired when a DELETE statement is
executed for this table:
CREATE TABLE test_trigger
(col1 int,
col2 char(6) )
GO
INSERT INTO test_trigger VALUES (1, 'First')
INSERT INTO test_trigger VALUES (2, 'Second')
INSERT INTO test_trigger VALUES (3, 'Third')
INSERT INTO test_trigger VALUES (4, 'Fourth')
This document is created with the unregistered version of CHM2PDF Pilot

INSERT INTO test_trigger VALUES (5, 'Fifth')

GO
CREATE TRIGGER delete_test
ON test_trigger AFTER DELETE
AS
PRINT 'You just deleted a row!'
GO

Now let's put the trigger to the test. What do you think will happen when the following statement is executed?
DELETE test_trigger
WHERE col1 = 0

If you execute this statement, you'll see the following message:

You just deleted a row!
(0 row(s) affected)

The message appears because the DELETE statement is a legal DELETE from the test_trigger table. The trigger is
fired once no matter how many rows are affected, even if the number of rows is 0! To avoid executing code that is
meaningless, it is not uncommon to have the first statement in a trigger check to see how many rows were affected.
We have access to the @@ROWCOUNT function, and if the first thing the trigger does is check its value, it will
reflect the number of rows affected by the data modification statement that caused the trigger to be fired. So we could
change the trigger to something like this:
ALTER TRIGGER delete_test
ON test_trigger AFTER DELETE
AS
IF @@ROWCOUNT = 0 RETURN
PRINT 'You just deleted a row!'

The trigger will still fire, but it will end almost as soon as it begins. You can also inspect the trigger's behavior if many
rows are affected. What do you think will happen when the following statement is executed?
DELETE test_trigger

A DELETE without a WHERE clause means that all the rows will be removed, and we get the following message:
You just deleted a row!

(5 row(s) affected)

Again, the trigger was fired only once, even though more than one row was affected.

If a statement affects only one row or is a transaction unto itself, the trigger will exhibit the characteristics of per-row
or per-transaction execution. For example, if you set up a WHILE loop to perform an UPDATE statement
repeatedly, an update trigger would execute each time the UPDATE statement was executed in the loop.

An after trigger fires after the data modification statement has performed its work but before that work is committed
to the database. Both the statement and any modifications made in the trigger are implicitly a transaction (whether or
not an explicit BEGIN TRANSACTION was declared). Therefore, the trigger can roll back the data modifications
that caused the trigger to fire. A trigger has access to the before image and after image of the data via the special
pseudotables inserted and deleted. These two tables have the same set of columns as the underlying table being
This document is created with the unregistered version of CHM2PDF Pilot

changed. You can check the before and after values of specific columns and take action depending on what you
encounter. These tables are not physical structures—SQL Server constructs them from the transaction log. In fact,
you can think of the inserted and deleted tables as views of the transaction log. For regular logged operations, a
trigger will always fire if it exists and has not been disabled. You can disable a trigger by using the DISABLE
TRIGGER clause of the ALTER TABLE statement.

You cannot modify the inserted and deleted pseudotables directly because they don't actually exist. As I mentioned
earlier, the data from these tables can only be queried. The data they appear to contain is based entirely on
modifications made to data in an actual, underlying base table. The inserted and deleted pseudotables will contain as
many rows as the insert, update, or delete statement affected. Sometimes it is necessary to work on a row-by-row
basis within the pseudotables, although, as usual, a set-based operation is generally preferable to row-by-row
operations. You can perform row-by-row operations by executing the underlying insert, update, or delete in a loop so
that any single execution affects only one row, or you can perform the operations by opening a cursor on one of the
inserted or deleted tables within the trigger.

Specifying Trigger Firing Order

As I mentioned earlier, in most cases you shouldn't manipulate the trigger firing order. However, if you do want a
particular trigger to fire first or fire last, you can use the sp_settriggerorder procedure to indicate this ordering. This
procedure takes a trigger name, an order value (FIRST, LAST, or NONE), and an action (INSERT, UPDATE, or
DELETE) as arguments. Setting a trigger's order value to NONE removes the ordering from a particular trigger.
Here's an example:
sp_settriggerorder delete_test, first, 'delete'

Note that you'll get an error if you specify an action that is not associated with the particular trigger—for example, if
the above statement had tried to make delete_test the first "update" trigger. You'll also get an error if you attempt any
of the following:
 Setting a trigger order for an instead-of trigger.
 Specifying an order for a trigger when a trigger already exists for that action in that position. (For example, if
there is already a first delete trigger for a table, you must set the order value of that trigger to NONE before
you can define a different trigger as the first delete trigger.)
 Attempting to give the same trigger both the first and last order value for the same action. One trigger can
have two different order values, but only if the trigger is defined for two different actions. You must execute
sp_settriggerorder multiple times to accomplish this. Here's an example:
CREATE TRIGGER delete_update_test
ON test_trigger AFTER DELETE, UPDATE
AS
PRINT 'You just deleted or updated a row!'
GO
EXEC sp_settriggerorder delete_update_test, first, 'delete'
EXEC sp_settriggerorder delete_update_test, last, 'update'

Information about trigger type and firing order is stored in the status column of the sysobjects table. Nine of the bits
are needed to keep track of this information. Three bits are used to indicate whether the trigger is an insert, update, or
delete trigger, three are used for the first triggers for each action, and three are used for the last triggers for each
action. Table 12-2 shows which bits correspond to which properties.

Table 12-2. Bits in sysobjects.status that indicate a trigger's functionality.

Bit Decimal Value Meaning
8 256 Delete Trigger
9 512 Update Trigger
This document is created with the unregistered version of CHM2PDF Pilot

10 1024 Insert Trigger

14 16384 First Delete Trigger
15 32768 Last Delete Trigger
16 65536 First Update Trigger
17 131072 Last Update Trigger
18 262144 First Insert Trigger
19 524288 Last Insert Trigger

To check to see whether a trigger has a particular functionality, you can decode the sysobjects.status value or use the
OBJECTPROPERTY function. There are nine property functions, which correspond to the nine bits indicated in
Table 12-2. For example, to check to see whether a trigger is the last update trigger for its table, you would execute
this:
SELECT OBJECTPROPERTY(object_id('delete_update_test'), 'ExecIsLastUpdateTrigger')

A value of 1 means that the trigger has this property, a value of 0 means it doesn't, and a value of NULL means you
typed something wrong or the object is not a trigger. However, to find all the properties that a trigger has would mean
executing this statement nine times, checking nine different property values. To simplify the process, I created a new
version of the sp_helptrigger procedure called sp_helptrigger2(available on the companion CD), which provides
ordering information for each of the triggers. Here's an example of its use:
EXEC sp_helptrigger2 test_trigger
PARTIAL RESULTS:
trigger_name IsUpdate UpdateOrd IsDelete DeleteOrd IsInsert InsertOrd
------------------ -------- --------- -------- ----------- -------- ---------
delete_test 0 n/a 1 Unspecified 0 n/a
delete_update_test 1 Last 1 First 0 n/a

The IsUpdate column has a 1 if the trigger is an update trigger, and it has a 0 otherwise. If the trigger is not an update
trigger, the order of the update trigger is n/a. If the trigger is an update trigger, values for the trigger order are First,
Last, and Unspecified. You can interpret the IsDelete and DeleteOrd columns in the same way, as well as the
IsInsert and InsertOrd columns. This procedure is available on the companion CD.

Rolling Back a Trigger

Executing a ROLLBACK from within a trigger is different from executing a ROLLBACK from within a nested
stored procedure. In a nested stored procedure, a ROLLBACK will cause the outermost transaction to abort, but the
flow of control continues. However, if a trigger results in a ROLLBACK (because of a fatal error or an explicit
ROLLBACK command), the entire batch is aborted.

Suppose the following pseudocode batch is issued from SQL Query Analyzer:
begin tran
delete....
update....
insert.... -- This starts some chain of events that fires a trigger
-- that rolls back the current transaction
update.... -- Execution never gets to here - entire batch is
-- aborted because of the rollback in the trigger
if....commit -- Neither this statement nor any of the following
-- will be executed
else....rollback
begin tran....
insert....
This document is created with the unregistered version of CHM2PDF Pilot

if....commit
else....rollback

GO -- batch terminator only

select... -- Next statement that will be executed is here

As you can see, once the trigger in the first INSERT statement aborts the batch, SQL Server not only rolls back the
first transaction but skips the second transaction completely and continues execution following the GO.

Misconceptions about triggers include the belief that the trigger cannot do a SELECT statement that returns rows and
that it cannot execute a PRINT statement. Although you can use SELECT and PRINT in a trigger, doing these
operations is usually a dangerous practice unless you control all the applications that will work with the table that
includes the trigger. Otherwise, applications not written to expect a result set or a print message following a change in
data might fail because that unexpected behavior occurs anyway. For the most part, you should assume that the
trigger will execute invisibly and that if all goes well, users and application programmers will never even know that a
trigger was fired.

Be aware that if a trigger modifies data in the same table on which the trigger exists, using the same operation
(INSERT, UPDATE, or DELETE) won't, by default, fire that trigger again. That is, if you have an UPDATE trigger
for the inventory table that updates the inventory table within the trigger, the trigger will not be fired a second time.
You can change this behavior by allowing triggers to be recursive. You control this on a database-by-database basis
by setting the option recursive triggers to TRUE. It's up to the developer to control the recursion and make sure that
it terminates appropriately. However, it will not cause an infinite loop if the recursion isn't terminated because, just like
stored procedures, triggers can be nested only to a maximum level of 32. Even if recursive triggers have not been
enabled, if separate triggers exist for INSERT, UPDATE, and DELETE statements, one trigger on a table could
cause other triggers on the same table to fire (but only if sp_configure 'nested triggers' is set to 1, as you'll see in a
moment).

A trigger can also modify data on some other table. If that other table has a trigger, whether that trigger also fires
depends on the current sp_configure value for the nested triggers option. If that option is set to 1 (TRUE), which is
the default, triggers will cascade to a maximum chain of 32. If an operation would cause more than 32 triggers to fire,
the batch will be aborted and any transaction will be rolled back. This prevents an infinite cycle from occurring. If your
operation is hitting the limit of 32 firing triggers, you should probably look at your design—you've reached a point at
which there are no longer any simple operations, so you're probably not going to be ecstatic with the performance of
your system. If your operation truly is so complex that you need to perform further operations on 32 or more tables to
modify any data, you could call stored procedures to perform the actions directly rather than enabling and using
cascading triggers. Although cascading triggers can be of value, overusing them can make your system a nightmare to
maintain.

Instead-of Triggers
SQL Server 2000 allows you create a second kind of trigger, called an instead-of trigger. An instead-of trigger,
rather than the data modification operation that fires the triggers, specifies the action to take. Instead-of triggers are
different from after triggers in several ways:
 You can have only one instead-of trigger for each action (INSERT, UPDATE, and DELETE).
 You cannot combine instead-of triggers and foreign keys that have been defined with CASCADE on a table.
For example, if Table2 has a FOREIGN KEY constraint that references Table1 and specifies CASCADE as
the response to DELETE operations, you will get an error message if you try to create an instead-of trigger
for DELETE on Table2. However, you can have instead-of triggers for INSERT or UPDATE. Similarly, if
you already have an instead-of trigger on Table2, you cannot alter the table to add a foreign key constraint
with the CASCADE action for the same data modification operation.
 Instead-of triggers can never be recursive, regardless of the setting of the recursive triggers database option.
For example, if an instead-of trigger is executed for INSERT into Table1 and the trigger does an INSERT
This document is created with the unregistered version of CHM2PDF Pilot

into Table1, the instead-of trigger is not processed. Instead, the INSERT is processed as if there were no
instead-of trigger for INSERT, and any constraints and after triggers will take effect.

Instead-of triggers are intended to allow updates to views that are not normally updateable. For example, a view that
is based on a join normally cannot have DELETE operations executed on it. However, you can write an instead-of
DELETE trigger. The trigger has access to the rows of the view that would have been deleted had the view been a
real table. The deleted rows are available in a worktable, which is accessed with the name deleted, just like for after
triggers. Similarly, in an UPDATE or INSERT instead-of trigger, you can access the new rows in the inserted table.

Here's a simple example that uses a Table1 and Table2 and builds a view on a join of these tables:
USE pubs
SET NOCOUNT ON
-- drop table Table1
CREATE TABLE Table1
(a int PRIMARY KEY,
b datetime default getdate(),
c varchar(10))

--drop table Table2

CREATE TABLE Table2
(a int
,message varchar(100))
GO

/* Insert 4 rows into Table1 */

INSERT INTO Table1(a) VALUES (1)
INSERT INTO Table1(a) VALUES (2)
INSERT INTO Table1(a) VALUES (3)
INSERT INTO Table1(a) VALUES (4)

/* Insert 6 rows into Table2 */

INSERT INTO Table2 VALUES (1, 'first row')
INSERT INTO Table2 VALUES (1, 'second row')
INSERT INTO Table2 VALUES (2, 'first row')
INSERT INTO Table2 VALUES (2, 'second row')
INSERT INTO Table2 VALUES (2, 'third row')
INSERT INTO Table2 VALUES (3, 'first row')
GO

/* Create a view based on a join of the tables

and then an INSTEAD-OF TRIGGER on the view
*/

CREATE VIEW join_view AS

SELECT Table1.a as a1, b, c,
Table2.a as a2, message
FROM Table1 join Table2
ON Table1.a = Table2.a
GO

CREATE TRIGGER DEL_JOIN

ON join_view
INSTEAD OF DELETE

AS
DELETE Table1
WHERE a IN (SELECT a1 FROM deleted)
DELETE Table2
WHERE a IN (SELECT a2 FROM deleted)
This document is created with the unregistered version of CHM2PDF Pilot

In this case, the view contained values from each table that could be used to determine which rows in the base table
needed to be removed.

In addition to views based on joins, another kind of view that has severe limitations on its updateability is a view
based on a UNION. Some UNION views are updateable, and I'll discuss them in Chapter 16 when we look at
partitioned views. But for views that don't meet the conditions for partitioning, direct updates might not be possible.

In the following example, I'll create a contacts list view in the pubs database consisting of the name, city, state, and
country of all the authors, stores, and publishers:
USE pubs
GO
CREATE VIEW contact_list
AS
SELECT ID = au_id, name = au_fname + ' ' + au_lname,
city, state, country = 'USA'
FROM authors
UNION ALL
SELECT stor_id, stor_name, city, state, 'USA'
FROM stores
UNION ALL
SELECT pub_id, pub_name, city, state, country
FROM publishers

I want to be able to insert a new contact into this list, and to do that I'll use an instead-of INSERT trigger. The
inserted table in the trigger will have values only for the columns included in the view, so all other columns in all three
tables will have to have default values or allow NULLS. The only column not meeting this requirement is the contract
column of the authors table, which is a bit column. I'll alter the column to give it a default value:
ALTER TABLE authors
ADD CONSTRAINT contract_dflt DEFAULT 0 FOR contract

The trigger must determine which base table should accept the row inserted into the contacts_list view. Author ID
values look very different from store or publisher ID values, so by looking for a hyphen, I can determine that the new
row is an author. To distinguish stores from publishers, I can notice that there is a CHECK constraint on the
underlying publishers table that requires new publisher ID values to start with the digits 99. (You can execute
sp_helpconstraint on the publishers table to see this constraint.) This means that I can't allow any stores to have an
ID that starts with 99, and if you can't limit yourself to this restriction, you might need a different solution than this
instead-of trigger.

The trickiest part of this instead-of trigger is that the view has concatenated the authors' first and last names into a
single name column, so the instead-of trigger must separate a single name into a two-part name. It uses the simplest
approach of looking for the first space in the name. Everything before the first space is considered the first name, and
everything after the first space is considered the last name.
CREATE TRIGGER Insert_Contact
ON contact_list
INSTEAD OF INSERT
AS
IF @@ROWCOUNT = 0 RETURN
IF (SELECT COUNT(*) FROM inserted) > 1 BEGIN
PRINT 'Only one row at a time can be inserted'
RETURN
END
-- check for a hyphen in the fourth position in the ID
IF (SELECT substring(ID,4,1) FROM inserted) = '-'
-- Inserting an author
INSERT into authors(au_id, au_fname, au_lname, city, state)
This document is created with the unregistered version of CHM2PDF Pilot

SELECT id, rtrim(substring(name, 1, charindex(' ',name) - 1)),

rtrim(substring(name, charindex(' ',name) + 1,
datalength(name) - charindex(' ',name))), city, state
FROM inserted
ELSE
-- Check for two nines at the beginning of the ID
IF (SELECT ID FROM inserted) like '99[0-9][0-9]'
-- Inserting a publisher
INSERT INTO publishers (pub_id, pub_name, city, state, country)
SELECT * FROM inserted
ELSE
-- Inserting a store
INSERT INTO stores(stor_id, stor_name, city, state)
SELECT id, name, city, state from inserted
RETURN

You can write similar instead-of triggers for updates and deletes.

Managing Triggers
The options available for managing triggers are similar to those for managing stored procedures. Here are some of the
options:
 You can see the definition of a trigger by executing sp_helptext <trigger name>.
 You can create a trigger WITH ENCRYPTION so that its text cannot be exposed.
 You can change a trigger's definition using ALTER TRIGGER so that it keeps its same internal ID number.
WARNING
If you ALTER a trigger that has had any firing order properties set, all such order values will be lost when the ALTER
is done. You must reexecute sp_settriggerorder procedures to reestablish the firing order.

In addition, a special procedure called sp_helptrigger is available. It takes the name of a table as an argument, and
optionally the name of a data modification action, and gives you the name and some of the properties for each trigger
on that table. However, as I mentioned earlier, sp_helptrigger does not tell you what order the trigger will be fired in,
so I created the sp_helptrigger2 procedure to do that.

Using Triggers to Implement Referential Actions

As I discussed in Chapter 6, SQL Server implements FOREIGN KEY (or REFERENCES) constraints. You can
define the constraint to either not allow changes that would break the relationship (NO ACTION) or to cascade any
changes in the referenced table into the referencing table. The NO ACTION option is the default; with this option
specified for your foreign key, if you attempt a data modification that would break a relationship—for example, if you
attempt to delete the primary key row while foreign key references exist—the data modification is disallowed and the
command is aborted. In other words, no action is taken—in fact, ANSI SQL-92 calls this the NO ACTION
referential action. Colloquially, NO ACTION is often referred to as RESTRICT, but technically this is incorrect.
ANSI uses RESTRICT for security and permissions purposes.
NOTE
A FOREIGN KEY constraint can of course be directed toward a UNIQUE constraint, not just a PRIMARY KEY
constraint. But there is no performance difference if the referenced table is declared using UNIQUE. In this section,
we'll refer only to PRIMARY KEY in the referenced table for simplicity.

Keep in mind that a FOREIGN KEY constraint affects two tables, not just one. It affects the referenced table as well
as the referencing table. While the constraint is declared on the referencing table, a modification that affects the
primary key of the referenced table checks to see whether the constraint has been violated. The change to the
referenced table will be either disallowed or cascaded to the referencing table, depending on how the FOREIGN
This document is created with the unregistered version of CHM2PDF Pilot

KEY constraint was defined.

As described in Chapter 6, the SQL-92 standard specifies four referential actions for modifying the primary key side
of a relationship: NO ACTION, SET NULL, SET DEFAULT, and CASCADE. The four referential actions have the
following characteristics. (These descriptions were given in Chapter 6, but I'll repeat them here for your convenience.)
 NO ACTION Disallows the action if the FOREIGN KEY constraint would be violated. This is the only
referential action implemented by SQL Server 7 for a declared FOREIGN KEY constraint, and it's the only
one that must be implemented in order for a product to claim to be "SQL-92 conformant."
 SET NULL Updates the referencing table so that the foreign key columns are set to NULL. Obviously, this
requires that the columns be defined to allow NULL.
 SET DEFAULT Updates the referencing table so that the foreign key columns are set to their DEFAULT
values. The columns must have DEFAULT values defined.
 CASCADE Updates the referencing table so that the foreign key columns are set to the same values that the
primary key was changed to, or deletes the referencing rows entirely if the primary key row is deleted.

A declared FOREIGN KEY constraint, rather than a trigger, is generally the best choice for implementing NO
ACTION. The constraint is easy to use and eliminates the possibility of writing a trigger that introduces a bug. In SQL
Server 2000, you can also use FOREIGN KEY constraints for the CASCADE action if you want to simply cascade
any changes. You can use triggers to implement either of the other referential actions not currently available with
declared FOREIGN KEY constraints.

You can also use a trigger to customize the referential actions in other ways. For example, you can customize error
messages to make them more informative than the generic messages sent with a violation of a declared constraint.
Also, since you cannot use the CASCADE option for FOREIGN KEY constraints that reference the same table, you
could use a trigger to implement a cascading action on a self-referencing table.

Recall that constraint violations are tested before triggers fire. If a constraint violation is detected, the statement is
aborted and execution never gets to the trigger (and the trigger never fires). Therefore, you cannot use a declared
FOREIGN KEY constraint to ensure that a relationship is never violated by the update of a foreign key and then also
use a trigger to perform a SET NULL or SET DEFAULT action when the primary key side of the same relationship
is changed. If you use triggers to implement SET NULL or SET DEFAULT behavior for changes to the referenced
table, you must also use triggers to restrict invalid inserts and updates to the referencing table. You can and should still
declare PRIMARY KEY or UNIQUE constraints on the table to be referenced, however. It's relatively easy to write
triggers to perform referential actions, as you'll see in the upcoming examples. For the sake of readability, you can still
declare the FOREIGN KEY constraint in your CREATE TABLE scripts but then disable the constraint using ALTER
TABLE NOCHECK so that it is not enforced. In this way, you ensure that the constraint still appears in the output of
sp_help <table> and similar procedures.
NOTE
Although you're about to see triggers that perform referential actions, you might want to consider another option
entirely. If you want to define constraints and still have update and delete capability beyond NO ACTION and
CASCADE, a good alternative is using stored procedures that exclusively perform the update and delete operations.
A stored procedure can easily perform the referential action (within the scope of a transaction) before the constraint
would be violated. No violation occurs because the corresponding "fix-up" operation is already performed in the
stored procedure. If an INSERT or UPDATE statement is issued directly (and you can prevent that in the first place
by not granting such permissions), the FOREIGN KEY constraint still ensures that the relationship is not violated.

Here's an example of using triggers to implement a referential action. Rather than simply implementing NO ACTION,
we want to implement ON DELETE SET NULL and ON UPDATE SET DEFAULT for the foreign key relationship
between titleauthor and titles. We must create the following three triggers (or suitable alternatives):
-- 1. INSERT and UPDATE trigger on referencing table.
-- Disallow any insert or update if the foreign key title_id
-- in the referencing table titleauthor does not match
-- the primary key title_id in the referenced table titles.
CREATE TRIGGER INS_UPD_titleauthor
This document is created with the unregistered version of CHM2PDF Pilot

ON titleauthor
FOR INSERT, UPDATE
AS
-- Do any rows exist in the inserted table that do not have
-- a matching ID in titles?
IF EXISTS
(SELECT * FROM inserted WHERE inserted.title_id NOT IN
(SELECT titles.title_id FROM titles) )
BEGIN
RAISERROR('No matching title found. Statement will be
aborted.', 16, 1)
ROLLBACK TRAN
END
GO

-- 2. If primary key in referenced table titles is changed,

-- update any rows in titleauthor to the DEFAULT value.
-- This implements ON UPDATE SET DEFAULT. Notice that this trigger could
-- be easily changed to set the column to NULL instead of DEFAULT,
-- which would implement ON UPDATE SET NULL.
CREATE TRIGGER UPD_titles
ON titles
FOR UPDATE
AS
DECLARE @counter int
IF UPDATE(title_id)
BEGIN
UPDATE titleauthor
SET titleauthor.title_id=DEFAULT
FROM titleauthor, deleted
WHERE titleauthor.title_id=deleted.title_id

SET @COUNTER=@@ROWCOUNT
-- If the trigger resulted in modifying rows of
-- titleauthor, raise an informational message
IF (@counter > 0)
RAISERROR('%d rows of titleauthor were updated to
DEFAULT title_id as a result of an update to titles table',
10, 1, @counter)
END
GO

-- 3. DELETE of the referenced table titles will set to NULL the

-- referencing titleauthor rows
CREATE TRIGGER DelCascadeTrig
ON titles
FOR DELETE
AS
DECLARE @counter int
UPDATE titleauthor
SET title_id = NULL
FROM titleauthor, deleted
WHERE titleauthor.title_id=deleted.title_id
SET @counter=@@ROWCOUNT

IF (@counter > 0)
RAISERROR('%d rows of titleauthor were set to a
NULL title_id as a result
of a delete to the titles table', 10, 1, @counter)
GO

NOTE
In order for these examples to run, you must drop or suspend the existing FOREIGN KEY constraints on titleauthor
that reference titles as well as FOREIGN KEY constraints on two other tables (sales and roysched) that reference
This document is created with the unregistered version of CHM2PDF Pilot

titles. In addition, you must modify the title_id column of titleauthor to allow NULLS, and you must re-create the
PRIMARY KEY constraint on the titleauthor table as a UNIQUE constraint so that NULLs in the title_id column
are allowed. (The code on the companion CD that corresponds to these examples does this.)

The following trigger is just an example of how you can use triggers to implement a cascade action. This trigger is
actually unnecessary because you can define a FOREIGN KEY constraint to cascade any changes. It is a cascading
update trigger on the titles (referenced) table that updates all rows in the titleauthor table with matching foreign key
values. The cascading update is tricky, which is why I'm presenting it. It requires that you associate both the before
and after values (from the inserted and deleted pseudotables) to the referencing table. In a cascading update, you by
definition change that primary key or unique value. This means that typically the cascading update works only if one
row of the referenced table is updated. If you change the values of multiple primary keys in the same update
statement, you lose the ability to correctly associate the referencing table. You can easily restrict an update of a
primary key to not affect more than one row by checking the value of the @@ROWCOUNT function, as follows:
CREATE TRIGGER UpdCascadeTrig1
ON titles
AFTER UPDATE
AS
DECLARE @num_affected int, @title_id varchar(11),
@old_title_id varchar(11)
SET @num_affected=@@ROWCOUNT
IF (@num_affected=0) -- No rows affected, so nothing to do
RETURN

IF UPDATE(title_id)
BEGIN
IF (@num_affected=1)
BEGIN
SELECT @title_id=title_id FROM inserted
SELECT @old_title_id=title_id FROM deleted
UPDATE titleauthor
SET title_id=@title_id
FROM titleauthor
WHERE titleauthor.title_id=@old_title_id
SELECT @num_affected=@@ROWCOUNT
RAISERROR ('Cascaded update in titles of Primary Key
from %s to %s to %d rows in titleauthor', 10, 1,
@old_title_id, @title_id, @num_affected)
END
ELSE
BEGIN
RAISERROR ('Cannot update multiple Primary Key values
in a single statement due to Cascading Update
trigger in existence.', 16, 1)
ROLLBACK TRANSACTION
END
END

Again, this trigger is shown just as an example. If you define a FOREIGN KEY constraint to cascade updates, you
can update multiple primary key values in a single update. But you should carefully evaluate whether this is something
you really want to do. The inability to update more than one primary key value at a time might save you from making a
mistake. You typically won't want to make mass updates to a primary key. If you declare a PRIMARY KEY or
UNIQUE constraint (as you should), mass updates usually won't work because of the constraint's need for
uniqueness. Although such an update is rare, you might want to set the value to some expression, such as multiplying
all Customer ID values by 1000 to renumber them. In a situation like this, the ability to cascade updates can be
useful.

Another situation in which you might want to update multiple primary keys in a single statement is if you have a
mail-order customer list that uses the customer phone number as the primary key. As phone companies add numbers
This document is created with the unregistered version of CHM2PDF Pilot

to accommodate fax machines, server remote access, and cellular phones, some area codes run out of possible
numbers, which leads to the creation of new area codes. If a mail-order company needs to keep up with such
changes, it must update the primary key for all customers who get new area codes. If you assume that the phone
number is stored as a character string of 10 digits, you must update the first three characters only. If the new area
code is 425, for example, your UPDATE statement would look something like this:
UPDATE customer_list
SET phone = '425' + substring(phone, 4, 7)
WHERE zip IN (list of affected zip codes)

This UPDATE would affect the primary key of multiple rows, and it would be difficult to write a trigger to implement
a cascade operation (perhaps into an orders table).

You might think that you can write a trigger to handle multiple updates to the primary key by using two cursors to step
through the inserted and deleted tables at the same time. However, this won't work. There's no guarantee that the
first row in the deleted table will correspond to the first row in the inserted table, so you cannot determine which old
foreign key values in the referencing tables should be updated to which new values.

Although it is logically the correct thing to do, you are not required to create a PRIMARY KEY or UNIQUE
constraint on a referenced table. (Unless, of course, you are going to actually declare a FOREIGN KEY constraint to
reference a column in the referenced table.) And there is no hard-and-fast requirement that the constraint be unique.
(However, not making it unique violates the logical relationship.) In fact, the "one row only" restriction in the previous
cascading trigger is a tad more restrictive than necessary. If uniqueness were not required on title_id, the restriction
could be eased a bit to allow the update to affect more than one row in the titles table, as long as all affected rows
were updated to the same value for title_id. When uniqueness on titles.title_id is required (as should usually be the
case), the two restrictions are actually redundant. Since all affected rows are updated to the same value for title_id,
you no longer have to worry about associating the new value in the referencing table, titleauthor—there is only one
value. The trigger looks like this:
CREATE TRIGGER UpdCascadeTrig2
ON titles
FOR UPDATE
AS
DECLARE @num_distinct int, @num_affected int,
@title_id varchar(11)
SET @num_affected=@@ROWCOUNT
IF (@num_affected=0) -- No rows affected, so nothing to do
RETURN

IF UPDATE(title_id)
BEGIN
SELECT @num_distinct=COUNT(DISTINCT title_id) FROM inserted
IF (@num_distinct=1)
BEGIN
-- Temporarily make it return just one row
SET ROWCOUNT 1
SELECT @title_id=title_id FROM inserted
SET ROWCOUNT 0 -- Revert ROWCOUNT back
UPDATE titleauthor
SET titleauthor.title_id=@title_id
FROM titleauthor, deleted
WHERE titleauthor.title_id=deleted.title_id
SELECT @num_affected=@@ROWCOUNT
RAISERROR ('Cascaded update of Primary Key to value in
titles to %d rows in titleauthor', 10, 1,
@title_id, @num_affected)
END
ELSE
BEGIN
This document is created with the unregistered version of CHM2PDF Pilot

RAISERROR ('Cannot cascade a multirow update that

changes title_id to multiple different values.', 16, 1)
ROLLBACK TRANSACTION
END
END

Using COUNT(DISTINCT title_id) FROM inserted ensures that even if multiple rows are affected, they are all set
to the same value. So @title_id can pick up the value of title_id for any row in the inserted table. You use the SET
ROWCOUNT 1 statement to limit the subsequent SELECT to only the first row. This is not required, but it's good
practice and a small optimization. If you don't use SET ROWCOUNT 1, the assignment still works correctly but every
inserted row is selected and you end up with the value of the last row (which, of course, is the same value as any
other row). Note that you can't use the TOP clause in a SELECT statement that assigns to a variable, unless the
assignment is done by a subquery. For example, you can do this:
DECLARE @v1 varchar(11)
SET @v1 = (SELECT TOP 1 au_id FROM authors)
PRINT @v1

but you can't do this:

DECLARE @v1 varchar(11)
SET @v1 = TOP 1 au_id FROM authors
PRINT @v1

It's not good practice to do a SELECT or a variable assignment that assumes that just one row is returned unless
you're sure that only one row can be returned. You can also ensure a single returned row by doing the assignment like
this:
SELECT @title_id=title_id FROM titles GROUP BY title_id HAVING
COUNT(DISTINCT title_id)=1

Because the previous IF allows only one distinct title_id value, you are assured that the preceding SELECT
statement returns only one row.

In the previous two triggers, you first determine whether any rows were updated; if none were, you do a RETURN.
This illustrates something that might not be obvious: a trigger fires even if no rows were affected by the update. Recall
that the plan for the trigger is appended to the rest of the statements' execution plan. You don't know the number of
rows affected until execution. This is a feature, not a bug; it allows you to take action when you expect some rows to
be affected but none are. Having no rows affected is not an error in this case. However, you can use a trigger to
return an error when this occurs. The previous two triggers use RAISERROR to provide either a message that
indicates the number of rows cascaded or an error message. If no rows are affected, we don't want any messages to
be raised, so we return from the trigger immediately.

Recursive Triggers
SQL Server allows triggers to recursively call themselves if the database option recursive triggers is set to TRUE.
This means that if a trigger modifies the table on which the trigger is based, the trigger might fire a second time. But
then, of course, the trigger modifies the table and fires the trigger again, which modifies the table and fires the trigger
again?and so on. This process does not result in an infinite loop because SQL Server has a maximum nesting depth of
32 levels; after the 32nd call to the trigger, SQL Server generates the following error. The batch stops executing, and
all data modifications since the original one that caused the trigger to be fired are rolled back.
Server: Msg 217, Level 16, State 1, Procedure up_rt1, Line 4
This document is created with the unregistered version of CHM2PDF Pilot

Maximum stored procedure, function, trigger, or view nesting level

exceeded (limit 32).

Writing recursive routines is not for the faint of heart. You must be able to determine what further actions to take
based on what has already happened, and you must make sure that the recursion stops before the nesting reaches its
limit. Let's look at a simple example. Suppose we have a table that keeps track of budgets for various departments.
Departments are members (or children) of other departments, so the budget for a parent department includes the
budgets for all its child departments. If a child department's budget increases or decreases, the change must be
propagated to the parent department, the parent's parent, and so forth. The following code creates a small budget
table and inserts three rows into it:
CREATE TABLE BUDGET
(dept_name varchar(30) not null,
parent_name varchar(30) null,
budget_amt money not null)
GO
INSERT INTO budget values ('Internal Training', 'Training', $10)
INSERT INTO budget values ('Training', 'Services', $100)
INSERT INTO budget values ('Services', NULL, $500)
GO

The following trigger is fired if a single row in the budget table is updated. If the department is not the highest-level
department, its parent name is not NULL, so you can adjust the budget for the parent department by the same
amount that the current row was just updated. (The amount of the change is equal to the new amount in the inserted
table minus the old amount in the deleted table.) That new update causes the trigger to fire again. If the next
department has a non-null parent, the update is applied again. The recursion stops when you get to the highest level
and the parent department is NULL. In that case, the trigger simply returns and all work is done.
CREATE TRIGGER update_budget
ON budget FOR update AS
DECLARE @rows int
SELECT @rows = @@ROWCOUNT
IF (@rows=0) RETURN
IF (@rows > 1) BEGIN
PRINT 'Only one row can be updated at a time'
ROLLBACK TRAN
RETURN
END
IF (SELECT parent_name FROM inserted) IS NULL RETURN
UPDATE budget
SET budget_amt = budget_amt + (SELECT budget_amt FROM inserted) -
(SELECT budget_amt FROM deleted)
WHERE dept_name = (SELECT parent_name FROM inserted)
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
SQL Server allows you to group statements together in a program unit that will execute automatically whenever a
data modification statement occurs. This code, stored on the server as a trigger, can be defined as either an after
trigger or an instead-of trigger. Triggers can maintain referential integrity in ways that constraints cannot, and you can
write triggers to enforce other business rules that you can't specify using constraints.

To fully understand how triggers affect the work done in your database, you also need to understand how transactions
work. SQL Server provides four different modes of transaction behavior, called isolation levels, which control what
data is available to other processes while you're working with that data. Programming effectively with Transact-SQL
requires that you understand transactional topics, such as when transactions will be committed and when they can be
rolled back. Since you'll likely be working in a multiuser environment, it is vital that you make the appropriate
concurrency and consistency choices to suit the isolation level of your environment.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 13

Special Transact-SQL
Operations: Working with
Cursors and Large Objects
A relational database such as Microsoft SQL Server is naturally set oriented. This means that a given statement, such
as SELECT, returns a set of results—often more than one row of data. On the other hand, most programming
languages and many applications tend to be ultimately record based. For example, in an application that displays a list
of customers, users scroll through the list and periodically drill down to get more detail about a customer and perhaps
make some modifications. Then they might proceed to the next customer and do the operation again, and so on, one
record at a time.

The incongruity between the set-oriented approach and the record-based approach is sometimes referred to as an
impedance mismatch. To deal with this mismatch, SQL Server provides a significant bridge between the two
models: cursors.

You can think of a cursor as a named result set in which a current position is always maintained as you move through
the result set. For example, as you visually scan names in a printed phone book, you probably run your finger down
the entries. Your finger acts like a cursor—it maintains a pointer to the current entry. You can think of the entries on
the current page of the phone book as the result set from your query. The action of moving your finger through the
listings on that page is similar to the action of a scrollable cursor—it moves forward and backward or up and down
at different times, but it always maintains a current position on a particular row.

SQL Server has different types of cursors that you can choose based on your scrolling requirements and on how
insulated you want the cursor to be from data changes made by others. Special cursor libraries in ODBC, OLE DB,
and DB-Library have network optimizations that prevent each row fetched in a cursor from incurring the overhead of
its own network conversation. Figure 13-1 shows how a cursor works.
This document is created with the unregistered version of CHM2PDF Pilot

34

Cursor Basics
To work with a cursor, you take the following steps:
1. Declare the cursor using the DECLARE statement. You choose a cursor based on how sensitive you want it
to be to the changes of others and on the level of scrolling you want. Then you specify the SELECT statement
that will produce the cursor's result set. Often, the rows in the result set will be explicitly sorted using the
ORDER BY clause. Most database systems that provide cursors allow you to move only forward through the
result set. SQL Server's scrollable cursors allow you to reposition the cursor to any row. They are also fully
updateable.
2. Open the cursor. You can think of this step as executing the DECLARE statement from step 1.
3. Fetch rows in the cursor. The cursor position moves within the result set to get the next row or the previous
row, to go back five rows, to get the last row, and so on. Typically, the FETCH statement is executed many
times—at least once for every row in the result set. (The other cursor control statements tend to be executed
only once per cursor.) In SQL Server, you typically check the system function @@FETCH_STATUS after
each fetch. A 0 value indicates that the fetch was successful. A -1 value indicates that there are no more
rows—that is, the fetch would move the cursor beyond the result set, either past the last row or before the
first row. A value of -2 indicates that the row no longer exists in the cursor—it was deleted from the base
table after the cursor was opened or it was updated in a way that no longer meets the criteria of the SELECT
statement that generated the result set.

Figure 13-1. SQL Server cursors bridge set-based and record-based models.
4. Update or delete the row of the table in which the cursor is positioned. This step is optional. Here's the
syntax:

UPDATE table SET column = value WHERE CURRENT OF cursor_name

DELETE table WHERE CURRENT OF cursor_name

After the cursor is positioned at a specific row, that row is usually updated or deleted. Instead of supplying a
This document is created with the unregistered version of CHM2PDF Pilot

WHERE clause that identifies a row in the table using values such as the primary key for the UPDATE or
DELETE statement, you identify the row to be operated on by specifying CURRENT, which is the row the
cursor points to. This type of operation is known as a positioned update (or a positioned delete).
5. Close the cursor. This ends the active cursor operation. The cursor is still declared, so you can reopen it
without having to declare it again.
6. Deallocate the cursor to remove a cursor reference. When the last cursor reference is deallocated, SQL
Server releases the data structures comprising the cursor. (We'll see later in this chapter that you can have
multiple references to the same cursor by using cursor variables.) Because internal data structures consume
memory in the procedure cache, you should use the DEALLOCATE statement to clean up after you're done
with the cursor. Think of DEALLOCATE as essentially "UNDECLARE"—the opposite of DECLARE, just
as CLOSE is the opposite of OPEN. Below is an example of a simple cursor that fetches forward through
the authors table and retrieves rows one by one. (For now, note that this is partly pseudocode—and highly
inefficient—because we don't want to get immersed in the details quite yet.)
DECLARE au_cursor CURSOR FOR
SELECT * FROM authors ORDER by au_lname

OPEN au_cursor

WHILE (more rows)

FETCH NEXT FROM au_cursor -- Do until end
CLOSE au_cursor
DEALLOCATE au_cursor

Of course, to get all the rows from the authors table, it would be a lot easier to issue the SELECT statement
directly, which would return all the rows in one set. Using a cursor to return a simple result set is
inappropriate, and this example serves only to illustrate the basic flow of cursor operations. We'll talk about
uses for cursors next.
This document is created with the unregistered version of CHM2PDF Pilot

34

Cursors and ISAMs

Before you proceed, you must understand something crucial about using cursors: you should never let cursors turn
SQL Server into a network ISAM.

You might recall from Chapter 1 that the initial grand plan for SQL Server called for it to be a higher-performance
back end for Ashton-Tate's dBASE IV. dBASE was really a record-oriented, ISAM-like data management system
(that is, it did sequential, row-by-row processing). It was not set based. At the time, SQL Server didn't use cursors,
which made a difficult task even more difficult. The original plan to make the two record-oriented and set-based
systems totally compatible and seamlessly interchangeable was doomed from the outset because of inherent
differences in the models. If the original SQL Server had featured the rich cursor model it has now, cursors would
have been heavily used to build the dBASE IV front end from the start. But it was probably better that the impedance
mismatch became obvious. It forced the SQL Server development team to reexamine its basic goals and dramatically
improve its plans. It also highlighted the importance of orienting yourself to work with sets of data, not individual
records as you would do with an ISAM. Had cursors existed then, they probably would have been misused, with the
result being a bad SQL Server front end.

Cursors can be an important tool when used prudently. However, because cursors are record oriented, developers
who are familiar with ISAM systems (such as IMS, dBASE, VSAM, or the Microsoft Jet database engine used in
Microsoft Access) are often tempted to use cursors to port an application from an ISAM system to SQL Server.
Such a port can be done quickly, but this is also one of the fastest ways to produce a truly bad SQL Server
application. In the basic cursor example shown previously, the operation to fetch each row of the authors table is
much like an ISAM operation on the authors file. Using that cursor is an order of magnitude less efficient than simply
using a single SELECT statement to get all authors. This type of cursor misuse is more common than you'd think. If
you need to port an ISAM application to SQL Server, do a deep port—that is, go back and look at the basic design
of the application and the data structures before you do the port. A shallow port—making SQL Server mimic an
ISAM—is appropriate only if you're one of those programmers who believes that there's never time to do the port
right but there's always time to do it over.

For example, even a modestly experienced SQL Server programmer who wants to show authors and their
corresponding book titles would write a single SELECT statement similar to the one below that joins the appropriate
tables. This SELECT statement is likely to yield subsecond response time even if all the tables are large, assuming that
appropriate indexes exist on the tables.
SELECT A.au_id, au_lname, title
FROM authors A
JOIN titleauthor TA ON (A.au_id=TA.au_id)
JOIN titles T ON (T.title_id=TA.title_id)
ORDER BY A.au_id, title

NOTE
When I refer to response time, I mean the time it takes to begin sending results back to the client application—in other
words, how long it takes until the first row is returned.

In the preceding example, all the join processing is done at the back end and a single result set is returned. Minimal
conversation occurs between the client application and the server—only the single request is received and all the
qualifying rows are returned as one result set. SQL Server decides on the most efficient order in which to work with
the tables and returns a single result set to satisfy the request.

An ISAM programmer who doesn't know about an operation such as a join would approach this problem by
This document is created with the unregistered version of CHM2PDF Pilot

opening the authors "file" (as an ISAM programmer would think of it) and then iterating for each author by scanning
the titleauthor "connecting file." The programmer would then traverse into the titles file to retrieve the appropriate
records. So rather than write the simple and efficient SQL join described above, the programmer might see cursors as
the natural solution and write the code shown below. This solution works in a sense: it produces the correct results.
But it is truly horrific in terms of its relative complexity to write and its performance compared to the set-based join
operation. The join would be more than 100 times faster than this batch.
DECLARE @au_id char(11), @au_lname varchar(40), @title_id char(6),
@au_id2 char(11), @title_id2 char(6), @title varchar(80)

DECLARE au_cursor CURSOR FOR

SELECT au_id, au_lname FROM authors ORDER BY au_id

DECLARE au_titles CURSOR FOR

SELECT au_id, title_id FROM titleauthor ORDER BY au_id

DECLARE titles_cursor CURSOR FOR

SELECT title_id, title FROM titles ORDER BY title

OPEN au_cursor
FETCH NEXT FROM au_cursor INTO @au_id, @au_lname

WHILE (@@FETCH_STATUS=0)
BEGIN
OPEN au_titles
FETCH NEXT FROM au_titles INTO @au_id2, @title_id

WHILE (@@FETCH_STATUS=0)
BEGIN
-- If this is for the current author, get
-- titles too
IF (@au_id=@au_id2)
BEGIN
OPEN titles_cursor
FETCH NEXT FROM titles_cursor INTO
@title_id2, @title

WHILE (@@FETCH_STATUS=0)
BEGIN
-- If right title_id, display the values
IF (@title_id=@title_id2)
SELECT @au_id, @au_lname, @title

FETCH NEXT FROM titles_cursor INTO

@title_id2, @title
END
CLOSE titles_cursor
END
FETCH NEXT FROM au_titles INTO @au_id2, @title_id
END

CLOSE au_titles
FETCH NEXT FROM au_cursor INTO @au_id, @au_lname
END
CLOSE au_cursor

DEALLOCATE titles_cursor
DEALLOCATE au_titles
DEALLOCATE au_cursor

Although this cursor solution is technically correct and is similar to the way ISAM processing would be written, if
This document is created with the unregistered version of CHM2PDF Pilot

your developers are writing SQL Server applications like this, stop them immediately and educate them. It might be
easy for a developer who is familiar with SQL to see that the join is much better, but a developer who is steeped in
ISAM operations might see the cursor solution as more straightforward even though it is more verbose. This is much
more than just a style issue. Even for this trivial example from pubs with only 25 rows of output, the join solution is
orders of magnitude faster. SQL Server is designed for set operations, so whenever possible you should perform set
operations instead of sequential row-by-row operations. In addition to the huge performance gain, join operations are
simpler to write and are far less likely to introduce a bug.

Programmers familiar with the ISAM model do positioned updates. That is, they seek (or position) to a specific row,
modify it, seek to another row, update it, and so forth. Cursors also offer positioned updates, but performance is far
slower than what you can achieve using a single UPDATE statement that simultaneously affects multiple rows.
Although the previous example showed a SELECT operation, you can also accomplish updates and deletes using
subquery and join operations. (For details, see Chapter 9.)

Problems with ISAM-Style Applications

The main problems with the ISAM style of writing applications for SQL Server are as follows:
 The ISAM-style operation makes excessive requests to the SQL Server. When cursor operations are done
in the client application, a huge increase in network traffic results. In the join solution example we just
examined, one request was made to the server: the SQL query doing the join. But in ISAM-style code, if
every OPEN and FETCH presents a command to the server, this simple example that produces fewer than
30 rows of output results in more than 1000 commands being sent to the SQL Server!

For the sake of illustration, the procedure is written entirely with Transact-SQL cursors. But cursors are
typically written from a client application, and almost every OPEN and every FETCH would indeed be a
separate command and a separate network conversation. The number of commands grows large because of
all the nested iteration that happens with the series of inner loops, as you'll see if you trace through the
commands. Although this procedure is admittedly convoluted, it's not unlike ones that exist in real
applications.
 It is almost always more expensive to use a cursor than it is to use set-oriented SQL to accomplish a given
task. ISAM's only form of update is the positioned update, but this requires at least two round-trips to the
server—one to position on the row and another to change it. In set-oriented SQL, the client application tells
the server that it wants to update the set of records that meet specified criteria. Then the server figures out
how to accomplish the update as a single unit of work. When you use a cursor, the client doesn't allow the
server to manage retrieving and updating records.

In addition, using a cursor implies that the server is maintaining client "state" information—such as the user's
current result set at the server, usually in tempdb—as well as consumption of memory for the cursor itself.
Maintaining this state unnecessarily for a large number of clients is a waste of server resources. A better
strategy for using SQL Server (or any server resource, in fact) is for the client application to get a request in
and out as quickly as possible, minimizing the client state at the server between requests. Set-oriented SQL
supports this strategy, which makes it ideal for the client/server model.
 The ISAM-style application example uses a conversational style of communicating with the server, which
often involves many network round-trips within the scope of a single transaction. The effect is that transactions
take longer. Transactions handled in this way can require seconds or minutes between the BEGIN
TRANSACTION and COMMIT TRANSACTION statements. These long-running transactions might work
fine for a single user, but they scale much too poorly for multiple users if any update activity occurs.

To support transactional consistency, the database must hold locks on shared resources from the time the
resources are first acquired within the transaction until the commit time. Other users must wait to access the
same resources. If one user holds these locks longer, that user will affect other users more and increase the
possibility of a deadlock. (For more on locking, see Chapter 14.)
This document is created with the unregistered version of CHM2PDF Pilot

34

Cursor Models
The subject of cursors confuses many people, at least in part because several cursor models are available and those
models have different options and capabilities. We can group these models into three categories: Transact-SQL
cursors, API server cursors, and client cursors.

Transact-SQL Cursors
You use Transact-SQL cursors within SQL typically in a stored procedure or in a batch that needs to do
row-by-row processing. Transact-SQL cursors use the familiar statements DECLARE CURSOR, OPEN cursor,
and FETCH. At times, Transact-SQL cursors have been referred to as "ANSI cursors" because their syntax and
capabilities fully implement—and go well beyond—the scrollable cursor functionality specified by ANSI SQL-92.
The ANSI specification states that scrollable cursors are read-only. This isn't the way applications work, of course,
so fortunately SQL Server allows scrollable cursors that can be updated. SQL Server provides backward scrolling,
relative and absolute positioning within the result set, and many options regarding sensitivity to changes made by
others, as well as positioned updates and deletes. Few other products implement even the ANSI specification for
scrollable, read-only cursors, let alone a cursor model as rich as SQL Server's. The cursor functionality of SQL
Server is without question the richest in the industry. The term "ANSI cursor" is rarely used anymore because it
implies a feature that is commonplace in the industry; the extensive cursor functionality is unique to SQL Server and
greatly exceeds that specified by ANSI.

API Server Cursors

The OLE DB provider for SQL Server, the ODBC driver for SQL Server, and the DB-Library programming
libraries have special cursor functions that are optimized for cursor operations and better network use between a
client application and a server application. For a high-performance application, it is important that there be no
excessive conversation between the client and the server. The hallmark of high-quality client/server computing is to
have the client issue a single, brief command and then have the server respond. If every row fetched in a cursor had to
be individually requested over the network and had to provide all the metadata to describe the row fetched,
performance would be abysmal.

The OLE DB provider and the ODBC driver that Microsoft provides for SQL Server fully implement these special
cursor commands, as does the DB-Library programming interface. Higher-level interfaces, such as RDO and ADO,
also implement these functions. These capabilities have the cumbersome names "ODBC server cursors," "DB-Library
server cursors," and "OLE DB server cursors," so they're all generally referred to in the documentation as "API server
cursors."

API server cursors enable smarter communication between the calling client application and the SQL Server back
end. Central to this is the interface's ability to specify a "fat" cursor. That is, instead of thinking of a cursor as pointing
to a single row, imagine the cursor pointing to multiple rows. Take, for example, an application that displays a list box
of 20 customers at a time. As the user scrolls through the list box, a FETCH requests the next 20 rows. Or perhaps it
fetches the previous 20 rows, the last 20 rows, or the 20 rows thought to be about three-fourths of the way through
the result set. The interface is provided by such function calls as SQLFetchScroll for ODBC and
IRowset::GetNextRows for OLE DB. These functions make sense within those programming environments, and they
are used instead of SQL statements such as DECLARE CURSOR and FETCH. These API functions allow options
and capabilities, such as defining the "width" of the cursor (the number of rows), that cannot be expressed using
Transact-SQL cursor syntax. Another important performance optimization of API server cursors is that metadata is
sent only once for the cursor and not with every FETCH operation, as is the case with a Transact-SQL cursor
This document is created with the unregistered version of CHM2PDF Pilot

statement.

Besides offering performance advantages (especially for operations across a network), API server cursors have
other capabilities that Transact-SQL cursors do not have. You can declare API server cursors on a stored procedure
if the procedure contains only one SELECT statement. You use API server cursors mostly in application code.
NOTE
Many APIs provide access to API server cursors, but in this chapter I'll look at them primarily in the context of
Transact-SQL cursor syntax, which is the most widely readable syntax. But when a capability is available only via
API server cursors, I'll point that out. The information on Transact-SQL cursors is also relevant to all forms of API
server cursors. After reading the information here, you should consult the programming reference for the specific API
you use to better understand the capabilities offered.

Client Cursors
If you consider that the job of a cursor is to set a position within a result set, it should be clear to you that this task
can be performed (although possibly less efficiently) within the client application with no assistance from the server.
SQL Server originally had no cursor support, and many application designers wrote their own client-side cursor
libraries to do this type of operation. While SQL Server 4.2 did not have cursor support in the engine, it offered a rich
set of cursor functions for DB-Library so that application programmers would not have to reinvent the wheel. These
functions were implemented entirely on the client. No magic was performed, and these functions did nothing that other
ISVs couldn't do. But the designers knew from the outset that they eventually wanted cursor support in the engine.
They designed the API so that the bulk of the work could eventually be executed remotely at the server and that a
new API or substantial changes to applications using the API would not be necessary.

Version 6 introduced scrollable cursor support directly in the engine. With this support, applications written using the
DB-Library version 4.2 cursor library typically worked without any changes. In fact, they worked quite a bit better
because of the server-side assistance that they got. Version 6 added an optimized ODBC driver that fully exploited
SQL Server's support for scrollable cursors. With database management systems other than SQL Server (most of
which do not have scrollable cursors), it's not unusual for an ODBC driver to take on these cursor functions, as
DB-Library used to for SQL Server. But with the ODBC driver that ships with SQL Server, cursor operations use
the back-end engine's capabilities.

With SQL Server, you tend to make use of cursors on the server. Because client cursors are more of a programming
convenience, we won't cover them in much depth. Instead, we'll focus on the server. But be aware that you can still
use the OLE DB or ODBC cursor library to do the cursor operations for you (using client cursors) instead of having
them processed at the server.

Some people consider client cursors obsolete now that server cursors are available. But client cursors can make a lot
of sense when properly used. For example, if you want your user interface to provide forward and backward scrolling
but you want to use the more efficient default result set rather than a server cursor, you need a local buffering
mechanism to cache the entire result set and provide the scrolling operations. You can write this mechanism yourself,
but client cursors already provide it. Also, the Internet has brought new issues and challenges for "stateless"
management of data. (That is, the connection to SQL Server is not persistent.) Client cursors can play a major part in
these solutions. Although the introduction of server cursors in version 6 might have made client cursors obsolete, the
Internet will probably drive future work into new and more sophisticated client-cursor solutions.

Default Result Sets

Do not fall into the habit of thinking that every result set is a cursor, at least of the type previously discussed, or you'll
inch toward the dark side—always using cursors. Earlier we said that you can think of a cursor as a named result set.
You can think of rows not returned in a cursor as the default result set or as just a result set. A default result set is
really not a cursor. You can think of it as a fourth type of cursor if you remember that no cursor functionality at the
This document is created with the unregistered version of CHM2PDF Pilot

server is involved.

A default result set is frequently referred to as a firehose cursor in discussions of the cursor capabilities of ODBC.
Within the SQL Server ODBC driver, you use the same function call, SQLFetchScroll, to process rows from SQL
Server whether they emanate from a cursor operation or simply from a default result set. With ODBC, if you request
a cursor that is forward-only and read-only, with a row set size of 1, the driver doesn't ask for a cursor at all. Instead,
it asks for a default result set, which is much more efficient than a cursor. The term "firehose" reflects the way that the
server blasts rows to the client as long as the client keeps processing them so that there is room in the network buffer
to send more results. In Chapter 3, we saw how results are sent from the server to the client, and this is exactly what
happens with a firehose cursor. The fact that the same ODBC functions work for both cursor and noncursor results is
a nice programming convenience, so I'll continue to use the term "firehose cursor." Just remember that from the SQL
Server engine's perspective, it is not a cursor.

API Server Cursors vs. Transact-SQL Cursors

Although this book is not about OLE DB or ODBC programming, cursors are intrinsically related to those
programming interfaces. This shouldn't be too surprising, since you know that a key motivation for including cursors in
SQL Server was to bridge the impedance mismatch between the set operations of SQL Server and the ISAM
operations of traditional programming languages. And traditional programming languages use ODBC or OLE DB (or
higher level interfaces that build on them, such as ADO). For more details about API server cursors, along with
programming examples, see Hitchhiker's Guide to Visual Basic and SQL Server, 6th Edition (Microsoft Press,
1998)by William (Bill) Vaughn. Bill's discussion is obviously centered on Visual Basic, but he covers cursor options
thoroughly enough that you can apply your understanding to programming in Visual C++. If you program with Visual
Basic, Bill's book is a must-read.

SQL Server provides some specialized cursor functions that are called only internally by the OLE DB provider, the
ODBC driver, or DB-Library. These functions look like stored procedures and have stubs in the sysobjects table, so
you might see their names there (for example, sp_cursor). If you trace the functions going to SQL Server (using SQL
Profiler or perhaps a network sniffer), you'll see these functions flowing from the client application. In fact, you can
execute them directly from ad hoc tools such as SQL Query Analyzer, although you should not call them directly from
your applications.

These functions are not really stored procedures; they are capabilities implemented directly in SQL Server. The code
is compiled into sqlservr.exe—it is not a stored procedure or an extended stored procedure. I'll refer to such
functions as "pseudo-stored procedures." The stubs in sysobjects make cursor execution mimic the execution of a
stored procedure so that there is no need for major changes to the tabular data stream (TDS) between client and
server. Remember that when server cursor support was implemented in version 6, a major goal was to have SQL
Server work seamlessly with applications that had been written to use the client cursor functions of DB-Library
version 4.2, so implementing cursors without changing the TDS was a clever idea. These functions are not intended,
or supported, for direct use by an application. They are invoked directly by the ODBC driver, the DB-Library
programming libraries, and the native OLE DB provider for SQL Server.

Transact-SQL cursors and API server cursors use the same code within SQL Server, although the Transact-SQL
cursors do not directly call the pseudo-stored procedures. Because the procedures exist only for the sake of remoting
the work from OLE DB, ODBC, and DB-Library, an SQL statement, which is already executing inside the engine,
has no need to call the routines in that way. While Transact-SQL and API server cursors execute the same code and
have many similarities, you should not think of them as alternative ways to do the same thing.
This document is created with the unregistered version of CHM2PDF Pilot

34

Appropriate Use of Cursors

When should you use cursors? A somewhat simplistic answer is, "Only when you have to." This is not a totally
facetious answer, because if you can think of a set-based solution to a problem and avoid using a cursor, you should
do that. Ordinarily, cursors should be near the bottom of your list of possible solutions. Remember that SQL Server is
a relational database, so it is inherently set based. As I mentioned earlier, if you overuse cursors, you turn SQL Server
into an ISAM-like product and severely cripple its power.

You should not interpret this warning as a blanket statement that you should never use cursors. Scrollable cursors
were, in fact, near the top of the new features wish list of many members of the SQL Server development team. More
important, cursors were near the top of the developer community's wish list. Cursors are extremely powerful and
provide SQL Server with capabilities that cannot be achieved with other products. So let's look at some situations in
which cursors provide a great benefit.

Row-by-Row Operations
Transact-SQL cursors are great for row-by-row operations with a stored procedure or a single batch. For example,
suppose a table has some financial data, and for each row of that data we want to perform one of several sets of
conversions on the data. Each set of conversions is written as a separate stored procedure, so we want to send the
data values from each row as parameters to the appropriate stored procedure. So, for example, if the value in the
balance column is less than 1000, we want to call a procedure called BasicConversions; if the balance value is
between 1000 and 10,000, we want to call a procedure called ExtendedConversions; and if the balance value is
greater than 10,000, we want to call AdvancedConversions. You might think you could process this operation with
CASE, but remember that in Transact-SQL CASE is an expression that returns a value; CASE doesn't allow you to
choose completely different actions to perform. With Transact-SQL cursors, we can step through each row of the
table, and depending on the value in the balance column for each row, we can conditionally determine which of the
three stored procedures to execute.

Transact-SQL's FETCH command is almost always done in a loop, and the cursor is used to perform row-by-row
processing on the entire result set. But you should not use Transact-SQL cursors across multiple batches. For
example, you shouldn't issue singular FETCH commands as their own batches. If your application fetches rows, does
some work, fetches more rows, and so on, you should use API server cursors, not Transact-SQL cursors.

Query Operations
Transact-SQL cursors are often used in conjunction with EXECUTE('string') to write and execute SQL statements
based on the results of some query. For example, suppose we want to GRANT permission to UPDATE data in
every table in a database to a particular group of users. SQL Server provides a built-in role in every database called
db_datawriter, but this allows all three data modification operations: INSERT, UPDATE, and DELETE. To allow
only UPDATE permission, we want to create a user-defined role called db_dataupdaters and GRANT UPDATE to
this role on all user tables. Using Transact-SQL cursors, we can use the following generic batch to do the entire
operation—without having to return the names of the tables to the calling client application, reformat them into the
correct commands, and send them back to the server. Instead, the entire operation is done at the server; only the
initial request is sent from the client application.
EXEC sp_addrole 'db_dataupdaters'
GO
DECLARE tables_curs CURSOR FOR
SELECT name FROM sysobjects
This document is created with the unregistered version of CHM2PDF Pilot

WHERE type='U' -- user table, not system table

OPEN tables_curs
DECLARE @tablename varchar(30), @output_msg varchar(80)
FETCH NEXT FROM tables_curs INTO @tablename
WHILE (@@FETCH_STATUS=0 )
BEGIN
EXEC ('GRANT UPDATE ON ' + @tablename
+ ' TO db_dataupdaters')
IF (@@ERROR=0)
SELECT @output_msg=
'UPDATE permission granted on table '
+ @tablename
ELSE
SELECT @output_msg=
'Failed to grant UPDATE permission on table '
+ @tablename + ' @@ERROR=' +
CONVERT(varchar, @@ERROR)
PRINT @output_msg
FETCH NEXT FROM tables_curs INTO @tablename
END
CLOSE tables_curs
DEALLOCATE tables_curs

Scrolling Applications
The need for Transact-SQL cursors was not the biggest motivation for adding cursors to SQL Server. Rather, the
crying need was for API server cursors to support "scrolling applications." Many of these applications originated from
ISAM applications or started out as single-user Microsoft FoxPro or dBASE applications (or they followed that
paradigm—you know the type).

For example, think of an address book of the type used in your e-mail program. The user opens the address book
and scrolls up or down to view the list of names. The user drags the scroll bar slider to the bottom and expects to be
positioned at the last record. Dragging the slider to the beginning of the list, the user expects to be repositioned at the
first record. Is this an appropriate use for a cursor? It depends on what you want to do with the information in the list.
Here are some examples of how you might think about the problem.

EXAMPLE 1

A cursor is unnecessary if the address book includes only a few hundred entries, if it will be read-only, or if you're
not concerned with sensitivity to changes in data (that is, you don't care whether the most up-to-date copy of the data
is available). After the user's list box is initially populated, a few new names might be added or the information might
change in some way. But for our purposes, it is unnecessary for the list box to exhibit dynamic behavior such as
names appearing, disappearing, or changing as the user scrolls through the list. In this case and with these
requirements, you can simply issue the query and get the results. You can easily buffer a few hundred rows of
information on the client, so there is no need for cursors.

EXAMPLE 2

An API server cursor is appropriate if the address book has 100,000 entries. The user will probably look at only a
few dozen entries and then quit. Should you select all 100,000 rows to populate the list box so that the user can see
only a few dozen? In most cases, the answer is no. Using a cursor is reasonable in this situation; you can use an API
server cursor.

Suppose the list box can display 20 names at a time. You can set the cursor width to 20 (a fat cursor) so that with
any fetch forward or backward you get one list box worth of data. Or you can fetch 60 or 80 rows at a time so that
the user can scroll by a few screenfuls within the application without having to ask the server for more rows. If the
This document is created with the unregistered version of CHM2PDF Pilot

user moves the scroll bar slider to the bottom of the list, you fetch the bottom 20 rows by using LAST. If the user
moves the slider to the top, you fetch the top 20 rows by using FIRST. And if the slider moves three-fourths of the
way down the list, you can do some quick division and scroll to that point by using ABSOLUTE n. (For example,
you can easily determine the total number of qualifying rows. If 100,000 rows are available and you want to scroll
approximately three-fourths of the way down, you can do a FETCH ABSOLUTE 75000 statement.)

If you have to provide this type of application with a large amount of data, using a cursor makes sense. Of course, it
might be better if the user is required to type in a few characters—that would allow the application to qualify the result
set of the cursor. If you know that the user is interested only in names starting with S, for example, you can
significantly reduce the size of the cursor by adding the appropriate criteria (for example, name LIKE ('S%') in the
cursor SELECT statement.

Choosing a Cursor
Although it might seem as if you have an overabundance of choices for your cursor model, the decision-making
process is fairly straightforward. Follow these guidelines to choose the appropriate cursor for your situation:
 If you can do the operation with a good set-oriented solution, do so and avoid using a cursor. (In ODBC
parlance, use a firehose cursor.)
 If you've decided that a cursor is appropriate and you'll be doing multiple fetches over the network (such as to
support a scrolling application), use an API server cursor. You'll use OLE DB, ODBC, RDO, DB-Library,
ADO, or another API, depending on your application needs.
 For the most part, avoid using client-side cursors and products or libraries that perform many cursor
operations in the client application. Such applications tend to make excessive requests to the server and use
the server inefficiently—the multitude of network round-trips makes for a slow, sluggish application.

Table 13-1 compares the ways cursors are declared and opened and how operations are performed in
Transact-SQL cursor statements; their rough equivalents among the pseudo-stored procedures used by the ODBC
driver and DB-Library; and the ODBC and OLE DB cursor methods.

Table 13-1. Equivalent Transact-SQL cursor statements, pseudo-stored procedures, ODBC cursor functions,
and OLE DB cursor methods.
Transact-SQL Cursor
Pseudo-Stored Procedure ODBC Cursor Function OLE DB Cursors
Statement
Set row set properties
OPEN such as
SQLSetStmtAttr DBPROP_OTHERINSER
(SQL_ATTR_CURSOR__ T, DBPROP_
TYPE) SQLSetStmtAttr OTHERUPDATEDELETE,
DECLARE/OPEN sp_cursoropen
(SQL_ATTR_CONCURR DBPROP_OWNINSERT,
ENCY) SQLExecDirect or or
SQLExecute DBPROP_OWNUPDATE
DELETE to control cursor
behaviors
SQLFetch or
FETCH sp_cursorfetch IRowset::GetNextRows
SQLExtendedFetch
IRowsetChange::SetData or
UPDATE/DELETE
sp_cursor SQLSetPos IRowsetChange::DeleteRow
(positioned)
s
CLOSE/DEALLOCATE sp_cursorclose SQLCloseCursor IRowset::Release

Cursor Membership, Scrolling, and Sensitivity to

This document is created with the unregistered version of CHM2PDF Pilot

Change
In addition to understanding the cursor model (that is, Transact-SQL or API server cursors), you must understand
some other key options that deal with the "membership" of the cursor and the behavior the cursor exhibits as you
scroll through the result set. We can divide these options into two categories:
 Cursor types The API server cursors usually specify cursor behavior by dividing them into four types: static,
keyset, dynamic, and forward-only.
 Cursor behaviors The ANSI SQL-92 standard defines the keywords SCROLL and INSENSITIVE to
specify the behavior of cursors. Some of the APIs also support defining a cursor's behavior using these terms.
SQL Server 2000 allows you to specify many more aspects of the cursor's behavior.

Transact-SQL cursors can use either the ANSI-specified syntax with INSENSITIVE and SCROLL options or the
full range of cursor types, which we call Transact-SQL Extended Syntax. Prior to SQL Server version 7, only the
ANSI syntax was available. However, because Transact-SQL Extended Syntax, which allows all the various options
available through API cursors, provides much greater control and flexibility, I'll use that syntax in most of my
examples. The ANSI specification does not define standards for several semantic issues related to sensitivity to
changes made by others. The issue of sensitivity to changes is further complicated in the client/server environment
because using a FETCH statement means fetching across the network. When you use the ANSI specification to
declare a cursor, the behavior characteristics are specified before the word CURSOR:

DECLARE cursor_name [INSENSITIVE] [SCROLL] CURSOR

FOR select_statement
[FOR {READ ONLY | UPDATE [OF column_list]}]

When implementing cursors, Microsoft engineers had to make a lot of decisions about subtle behavioral issues. They
wanted efficient scrollable cursors that could be used for updating in a networked client/server environment.
However, according to the ANSI specification, scrollable cursors are read-only! Fortunately, they were able to go
way beyond the specification when they implemented scrollable, updateable cursors. In fact, Microsoft's original
cursor specification went so far beyond the original ANSI specification that they started using the terminology of the
four cursor types (static, keyset, dynamic, and forward-only) even though the syntax allowed only the words
INSENSITIVE and SCROLL. Since the new API-based syntax for declaring cursors actually uses these types in the
declaration, that is the syntax I'll mainly use.

Here's how to declare a cursor using Transact-SQL Extended Syntax:

DECLARE cursor_name CURSOR

[LOCAL | GLOBAL]
[FORWARD_ONLY | SCROLL]
[STATIC | KEYSET | DYNAMIC | FAST_FORWARD]
[READ_ONLY | SCROLL_LOCKS | OPTIMISTIC]
[TYPE_WARNING]
FOR select_statement
[FOR UPDATE [OF column_list]]

The keywords describe how membership in the cursor is maintained (that is, which rows qualify), the available
scrolling capabilities, and the cursor's sensitivity to change. Notice that the keywords appear after the word CURSOR
in the DECLARE statement. That is how the parser distinguishes the older ANSI cursor syntax from the new
Transact-SQL Extended Syntax for cursors: in the former, the cursor qualifiers come before the word CURSOR; in
the latter, the qualifiers come after.

Static Cursors
A static cursor is attached to a snapshot of the data that qualifies for the cursor. The snapshot is stored in tempdb. A
This document is created with the unregistered version of CHM2PDF Pilot

static cursor is read-only, and the rows in the cursor and the data values of the rows never change when you fetch
anywhere in the cursor because you operate on a private, temporary copy of the data. Membership in a static cursor
is fixed—that is, as you fetch in any direction, new rows cannot qualify and old rows cannot change in such a way that
they no longer qualify.

Before you use a static cursor, consider whether you need to use a cursor at all. If the data is static and read-only, it
probably makes more sense to process it all in the client application as a default result set and not use a cursor. If the
number of rows is too large to reasonably process on the client, creating this temporary table on the server can also
prove to be an expensive undertaking.

If you use ANSI-style cursors and specify the modifier INSENSITIVE, the cursor will be static. Using
Transact-SQL Extended Syntax, you can actually specify the keyword STATIC. A private temporary table is created
for the SELECT statement you specify. Note that the cursor cannot do positioned updates or deletes on this snapshot
of data in the temporary table. For comparison, here are two cursor definitions for the same query, using the two
syntax possibilities:
-- ANSI-style syntax for declaring static cursor
DECLARE my_cursor1 INSENSITIVE CURSOR
FOR SELECT au_id, au_lname
FROM authors
WHERE state != 'CA'

-- Transact-SQL Extended Syntax for declaring static cursor

DECLARE my_cursor1 CURSOR STATIC
FOR SELECT au_id, au_lname
FROM authors
WHERE state != 'CA'

The result rows returned from the two cursors above are identical, with one minor behavioral difference. A static
cursor defined using the ANSI syntax and the keyword INSENSITIVE is not scrollable—that is, you can fetch rows
only in the forward direction. If you want the cursor to be scrollable, you must specify the SCROLL keyword. The
Transact-SQL Extended Syntax creates static cursors that are scrollable by default.

You might want to use a static cursor for some "what if"–type operations on the data when you know that the
changes will never be reflected in the base tables. If you want to carry out this type of operation, you can do a
SELECT INTO in a temporary table and then declare a cursor on that table without specifying STATIC. If you
specify a query that performs an aggregate function (such as SUM, MIN, MAX, GROUP BY, or UNION), the
cursor might also have to be materialized in a temporary table and therefore will automatically be a static cursor.

Keyset Cursors
With a keyset cursor, a list of all the key values for the rows that meet the SELECT statement criteria is kept in
tempdb. For example, if you declare a scrollable cursor on the customer table, the keyset is a list of those cust_id
values that qualified for membership in the SELECT statement when the cursor was opened. Suppose that when the
cursor was opened, the SELECT statement used to declare the cursor had a WHERE clause in this form:
WHERE cust_balance > 100000

Suppose also that customers 7, 12, 18, 24, and 56 (that is, rows with cust_id of those values) qualified. These keys
are used whenever fetching is performed. Conceptually, further selection for the cursor takes this form
WHERE cust_id IN (7, 12, 18, 24, 56)

rather than this form:

This document is created with the unregistered version of CHM2PDF Pilot

WHERE cust_balance > 100000

(Internally, the SELECT statement is not issued again. This is for purposes of illustration only.) Membership in the
keyset cursor is fixed. That is, these five identified rows are part of the cursor and no other rows are seen in
subsequent fetching. Even if other rows that meet the SELECT statement criteria are subsequently inserted or
updated, they are not seen by this cursor after it is opened.

In a keyset cursor as opposed to a static cursor, you can see changes to the data in the rows that meet the SELECT
criteria when the cursor is opened. For example, if another user modifies the customer balance while you hold the
cursor open and then fetch the row again, you see those changes (unless you hold locks to prevent such changes). In
fact, even if the modification causes the row to no longer satisfy the criteria of cust_balance > 100000, you still see
the row in a subsequent fetch. The row disappears only if it is actually deleted. The key value still exists because it
was squirreled away in tempdb, but the row is gone. Default values, NULL values, blank spaces, or zeros (as
appropriate) are supplied for the column values. But more important, the @@FETCH_STATUS system function
returns a value of -2 in this case, indicating that the row no longer exists.

Because the keys are stored in tempdb when the cursor is open and membership is fixed, keyset cursors can fetch to
an absolute position within the result set. For example, it is reasonable to fetch to row 4 of the cursor:
FETCH ABSOLUTE 4 FROM cursor_name

It shouldn't come as a surprise that a keyset cursor demands that a unique index exist on every table used in the
SELECT statement for the cursor. The unique index is necessary to identify the keys. The index can be created
directly (with CREATE INDEX), or it can exist as a result of a Primary Key or Unique constraint.
NOTE
In SQL Server 2000, every clustered index is treated as unique internally, even if you didn't specifically define the
index to be unique. For this reason, you can always define a keyset cursor on a table that has a clustered index.

If you declare a keyset cursor using Transact-SQL Extended Syntax on a table that doesn't have a unique index, you
do not receive an error message or any warning unless the cursor was declared with the option TYPE_WARNING.
Without that option specified, you just don't get a keyset cursor. Instead, the cursor is created as STATIC. Later in
this chapter, we'll see the stored procedures that provide information about your currently available cursors.

Dynamic Cursors
You can think of a dynamic cursor as a cursor in which the SELECT statement is applied again in subsequent
FETCH operations. That is, the cursor does not refetch specific rows, as in this statement:
WHERE cust_id IN (12, 18, 24, 56, 7)

Instead, conceptually, the WHERE clause is reapplied. For example:

WHERE cust_balance > 100000

This means that membership is not fixed—subsequent fetches might include newly qualifying rows, or previously
qualifying rows might disappear. This can occur because of changes you have made within the cursor or because of
changes made by others. If the cursor has not locked the rows of the result set (which is dependent on the
concurrency options selected, as we'll see in a moment), the changes made by others are seen in your cursor when
you do a subsequent fetch on the cursor. You see such changes only on a subsequent fetch. The image of data in the
buffer from the last fetch is just that—a copy in memory of what the row or rows looked like when the last fetch was
performed.
This document is created with the unregistered version of CHM2PDF Pilot

Since membership in a dynamic cursor is not fixed, there is no guarantee that subsequent fetches will always bring up
the same information. For this reason, FETCH ABSOLUTE is not supported for cursors declared using the
DYNAMIC keyword, even if you declare the cursor through the API. It doesn't make sense, for example, to simply
fetch to row 104 if every subsequent fetch brings up a row 104 that contains different information. However, FETCH
RELATIVE is supported with dynamic cursors, which sometimes surprises people. When used with a dynamic
cursor, FETCH RELATIVE n starts fetching from the first row in the current cursor set and skips the first n rows
from that point.

A cursor declared with an ORDER BY clause can be dynamic only if an index contains keys that match the ORDER
BY clause. If no such index exists, the cursor automatically converts to either a keyset or a static cursor. As
previously mentioned, you need a unique index on some column in the table to get a keyset cursor. So if you have a
unique index that just doesn't happen to be on the columns in the ORDER BY, you get a keyset cursor. If there is no
index that matches the ORDER BY and no unique index on any column, the cursor reverts to a static cursor.

Forward-Only Cursors
Forward-only cursors are dynamic cursors that allow only a FETCH type of NEXT. It's okay for you to think of
SQL Server as having only three types of cursors (static, keyset, and dynamic). Forward-only is treated here as its
own type because the Transact-SQL Extended Syntax, as well as API server cursors, specify forward-only at the
same level as KEYSET and DYNAMIC.

Forward-only scrolling is consistent with the recommended use of Transact-SQL cursors to provide row-by-row
processing within a stored procedure or batch. Such processing is usually from start to end—one way—so that rows
are never refetched. If you use Transact-SQL cursors appropriately, you typically scroll forward-only to do a
row-by-row operation and you do not revisit rows. Forward-only cursors are usually the fastest type of cursor, but a
standard SELECT statement (not a cursor) is still significantly faster.

Fast Forward-Only Cursors

Transact-SQL Extended Syntax allows for a special type of forward-only cursor called a fast forward-only cursor.
This type of cursor is always read-only, in addition to being forward-only. You can also specify fast forward-only
cursors through the ODBC driver, and in fact this is the environment in which they provide the most benefit.
Applications using the SQL Server ODBC driver can set the driver-specific statement attribute
SQL_SOPT_SS_CURSOR_OPTIONS to SQL_CO_FFO or SQL_CO_FFO_AF. The SQL_CO_FFO_AF
option specifies that an autofetch option also be enabled. Autofetch enables two optimizations that can significantly
reduce network traffic:
 When the cursor is opened, the first row or batch of rows is automatically fetched from the cursor. This saves
you from having to send a separate fetch request across the network.
 When a fetch hits the end of the cursor, the cursor is automatically closed. This saves you from having to send
a separate close request across the network.

The most dramatic improvement comes when you process cursors with relatively small result sets that can be cached
in the memory of an application. The fast forward-only cursor with autofetch enabled is the most efficient way to get a
result set into an ODBC application.

The autofetch option is not available when you use Transact-SQL Extended Syntax. It is meant as an optimization
when you declare and manage cursors through your client application.

Implicit conversion of fast forward-only cursors Fast forward-only cursors are implicitly converted to other
cursor types in certain situations. If the SELECT statement joins one or more tables with triggers to tables without
triggers, the cursor is converted to a static cursor. If a fast forward-only cursor is not read-only, it is converted to a
dynamic cursor. If the SELECT statement is a distributed query that references one or more remote tables on linked
servers, the cursor is converted to a keyset-driven cursor.
This document is created with the unregistered version of CHM2PDF Pilot

Other implicit conversions are also carried out when you work with Transact-SQL Extended Syntax cursors,
depending on the structure of the underlying tables and the keywords used in the cursor's SELECT statement. We
saw previously that a cursor declared as a keyset is implicitly converted to a static cursor if any of the underlying
tables don't have a unique index. A cursor based on a SELECT statement that needs to build a temp table is implicitly
converted to a static cursor. If you have any doubt whether SQL Server actually created the type of cursor you
requested, you can use the cursor procedures described in the following sections to verify all of the cursor's
properties.
This document is created with the unregistered version of CHM2PDF Pilot

34

Working with Transact-SQL

Cursors
In the following sections, I'll discuss the cursor syntax for each of the cursor manipulation statements and then look at
issues of locking and concurrency control. Throughout this discussion, we'll look at cursor functionality that is present
in the server, which in some cases is available only through API server cursors.

DECLARE
Here again is the Transact-SQL Extended Syntax for declaring a cursor with a given name:

DECLARE cursor_name CURSOR

[LOCAL | GLOBAL]
[FORWARD_ONLY | SCROLL]
[STATIC | KEYSET | DYNAMIC | FAST_FORWARD]
[READ_ONLY | SCROLL_LOCKS | OPTIMISTIC]
[TYPE_WARNING]
FOR select_statement
[FOR UPDATE [OF column_list]]

A cursor is always limited in scope to the connection that declares it. Although the cursor is named, it is not visible to
other connections and other connections can have cursors of the same name. This is true whether the cursor is
declared as GLOBAL or LOCAL; it is always private to the connection.

Global and Local Cursors

By default, Transact-SQL cursors are global for the connection in which they are declared. This means that you can
declare a cursor in one batch and use it in subsequent batches or fetch each row in its own batch. It also means that
you can declare a cursor in a stored procedure and have access to the cursor outside of the stored procedure. This is
not necessarily a good thing, because if the stored procedure is then called a second or subsequent time, it can get an
error message when it tries to declare a cursor that already exists (from a previous execution using the same
connection).

You can use the syntax for declaring a cursor to specify whether the cursor is global or local. As mentioned earlier,
global means global to the connection for which the cursor was declared. A local cursor is local to the batch, stored
procedure, or trigger in which it is declared, and no reference can be made to the cursor outside of that scope. If you
need access to a cursor declared as local inside a stored procedure, you can pass the cursor as an output parameter
to the batch that called the procedure. You'll see the details of this when we look at cursor variables and parameters.

Transact-SQL cursors are global by default in SQL Server. You can override this at the database level by setting the
database option default to local cursor to TRUE or by using the ALTER DATABASE command to set the
CURSOR_DEFAULT option to LOCAL. Local cursors are implicitly deallocated when the batch, stored procedure,
or trigger terminates unless it was passed back in an output parameter. If it is passed back in an output parameter, the
cursor is deallocated when the last variable referencing it is deallocated or goes out of scope.

Specifying the Cursor's Result Set

This document is created with the unregistered version of CHM2PDF Pilot

A Transact-SQL cursor always has its result set generated from a SELECT statement. The SELECT statement can
be ordered (ORDER BY), and it often is. An API server cursor can also be declared; the result set is the result of a
stored procedure. This powerful capability, which is important for scrolling applications, is not available using the
Transact-SQL syntax.

Specifying the Cursor's Behavior

In addition to specifying whether the cursor is local or global, the DECLARE statement also specifies what kind of
directional movement is possible, using the keywords SCROLL and FORWARD_ONLY. When neither of these is
specified, FORWARD_ONLY is the default, unless a STATIC, KEYSET, or DYNAMIC keyword is specified.
STATIC, KEYSET, and DYNAMIC cursors default to SCROLL.

The DECLARE statement also controls how much of the cursor's result set is stored in tempdb and how subject to
change the cursor's result set is. These behaviors are specified using the STATIC, KEYSET, and DYNAMIC
keywords, which I discussed earlier. The DECLARE statement uses the READ_ONLY option to prevent updates
from being made through this cursor by the connection that opened the cursor. A READ_ONLY cursor cannot be
referenced in a WHERE CURRENT OF clause in an UPDATE or DELETE statement. This option overrides the
cursor's default ability to be updated.

The READ_ONLY option controls only whether the connection that opened the cursor can update the data through
the cursor. It has no affect on what changes can be made to the cursor's data from other connections. Such changes
are controlled by the STATIC, KEYSET, and DYNAMIC options. The READ_ONLY option also controls the
locks that are taken when rows are fetched through the cursor. The locking (or concurrency) options are discussed
later.

It is worth pointing out that if the DECLARE statement includes a local variable, the variable's value is captured at
declare time, not at open time. In the following example, the cursor uses a WHERE clause of qty > 30, not qty > 5:
DECLARE @t smallint
SELECT @t=30

DECLARE c1 CURSOR SCROLL FOR SELECT ord_num, qty FROM sales

WHERE qty > @t

DECLARE @ordnum varchar(20), @qty smallint

SELECT @t=5
OPEN c1

OPEN
The OPEN statement opens the cursor, generating and populating temporary tables if needed:

OPEN cursor_name

If the cursor is keyset based, the keys are also retrieved when the cursor is opened. Subsequent update or fetch
operations on a cursor are illegal unless the cursor opens successfully. You can call the system function
@@CURSOR_ROWS after the OPEN statement to retrieve the number of qualifying rows in the last opened
cursor. Depending on the number of rows expected in the result set, SQL Server might populate the keyset cursor
asynchronously on a separate thread. This allows fetches to proceed immediately, even if the keyset cursor is not fully
populated. Table 13-2 shows the values returned by using @@CURSOR_ROWS and a description of each value.

Table 13-2. Possible values returned by @@CURSOR_ROWS.

This document is created with the unregistered version of CHM2PDF Pilot

Value Description
The cursor is dynamic. @@CURSOR_ROWS is not
-1
applicable.
A keyset cursor is still in the process of being populated
asynchronously. This value refers to the number of rows in
the keyset so far. The negative value indicates that the
-m
keyset is still being retrieved. A negative value
(asynchronous population) can be returned only with a
keyset cursor.
n This value refers to the number of rows in the cursor.
No cursors have been opened, the last opened cursor has
0
been closed, or the cursor has no rows.

To set the threshold at which SQL Server generates keysets asynchronously (and potentially returns a negative value),
you use the cursor threshold configuration option with sp_configure. Note that this is an "advanced option" and is
not visible unless Show Advanced Options is also enabled with sp_configure. By default, the setting for this cursor
threshold is -1, which means that the keyset is fully populated before the OPEN statement completes. (That is, the
keyset generation is synchronous.) With asynchronous keyset generation, OPEN completes almost immediately and
you can begin fetching rows while other keys are still being gathered. However, this can lead to some puzzling
behaviors. For example, you can't really FETCH LAST until the keyset is fully populated. Unless you are generating
large cursors, it's usually not necessary to set this option—synchronous keyset generation works fine and the behavior
is more predictable. If you need to open a large cursor, however, this is an important capability for maintaining good
response times.

FETCH
The FETCH statement fetches a row from the cursor in the specified direction:

FETCH [row_selector FROM] cursor_name [INTO @vl, @v2, ...]

A successful fetch results in the system function @@fetch_status returning the value 0. If the cursor is scrollable,
row_selector can be NEXT, PRIOR, FIRST, LAST, ABSOLUTE n, or RELATIVE n. (We've seen one exception:
a DYNAMIC cursor cannot FETCH ABSOLUTE.) If no row_selector is specified, the default is NEXT. FETCH
NEXT operations are by far the most common type of fetch for Transact-SQL cursors. This is consistent with their
recommended use when row-by-row processing is needed with a stored procedure or batch. Such processing is
nearly always from start to finish.

More exotic forms of fetch are also supported. ABSOLUTE n returns the nth row in the result set, with a negative
value representing a row number counting backward from the end of the result set. RELATIVE n returns the nth row
relative to the last row fetched, and a negative value indicates the row number is counted backward starting from the
last row fetched. Fetches using NEXT, PRIOR, or RELATIVE operations are done with respect to the current
cursor position.

RELATIVE 0 means "refetch the current row." This can be useful to "freshen" a "stale" row. The row number n
specified in RELATIVE and ABSOLUTE fetch types can be a local variable or a parameter of type integer,
smallint, or tinyint.

When a cursor is opened, the current row position in the cursor is logically before the first row. This causes the fetch
options to have slightly different behaviors if they are the first fetch performed after the cursor is opened:
 FETCH NEXT is equivalent to FETCH FIRST.
 FETCH PRIOR does not return a row.
 FETCH RELATIVE is equivalent to FETCH ABSOLUTE if n is positive. If n is negative or 0, no row is
This document is created with the unregistered version of CHM2PDF Pilot

returned.

The ANSI SQL-92 specification states that scrollable cursors must be read-only. There really isn't any rational reason
for this, except that updateable scrollable cursors were considered too hard to implement to put into the standard.
Fortunately, SQL Server scrollable cursors are indeed updateable—a quantum leap beyond what the standard
specifies!

If the cursor is not scrollable, only NEXT (or no explicit parameter, which amounts to the same thing) is legal. When
the cursor is opened and no fetch is done, the cursor is positioned before the first row—that is, FETCH NEXT
returns the first row. When some rows are fetched successfully, the cursor is positioned on the last row fetched.
When a FETCH request causes the cursor position to exceed the given result set, the cursor is positioned after the
last row—that is, a FETCH PRIOR returns the last row. A fetch that causes the position to go outside the cursor
range (before the first row or after the last row) causes the system function @@FETCH_STATUS to return the value
-1.
TIP
Probably the most common and appropriate use of Transact-SQL cursors is to FETCH NEXT within a loop while
@@FEtch_status equals 0 (or <> -1). However, you must issue the first FETCH before checking the value of
@@FETCH_STATUS and then reissue your FETCH within the loop. If your cursor is a keyset cursor, you must
also check for a @@FETCH_STATUS of-2 after every row is read to verify that the row is still available. If a row is
fetched but no longer exists with a keyset cursor, the row is considered "missing" and @@FETCH_STATUS returns
-2. The row might have been deleted by some other user or by an operation within the cursor. A missing row has
been deleted, not simply updated so that it no longer meets the criteria of the SELECT statement. (If that's the
semantic you want, you should use a dynamic cursor.)

Missing rows apply to keyset cursors only. A static cursor, of course, never changes—it is merely a snapshot of the
data stored in a temporary table and cannot be updated. A dynamic cursor does not have fixed membership. The row
is not expected to be part of a dynamic cursor because rows come and go as the cursor is fetched, depending on
what rows currently meet the query criteria. So there is no concept of a missing row for a dynamic cursor.

If an INTO clause is provided, the data from each column in the SELECT list is inserted into the specified local
variable; otherwise, the data is sent to the client as a normal result set with full metadata using TDS. In other words,
the result set of a Transact-SQL cursor is the same as any other result set. But doing 50 fetches back to the
application in this way produces 50 result sets, which is inefficient. An error occurs from the fetch if the datatypes of
the column being retrieved and the local variable being assigned are incompatible or if the length of the local variable is
less than the maximum length of the column. Also, the number of variables and their order must exactly match those of
the selected columns in the SELECT statement that defines the cursor.

UPDATE
The UPDATE statement updates the table row corresponding to the given row in the cursor:

UPDATE table_name SET assignment_list WHERE CURRENT OF cursor_name

This is referred to as a positioned update. Using Transact-SQL cursors, a positioned update (or a positioned
delete) operates only on the last row fetched and can affect only one row. (API server cursors can update multiple
rows in a single operation. You can think of this capability as an array of structures that can all be applied in one
operation. This is a significant capability, but it makes sense to use it only from an application programming language.)

An update involving a column that is used as a key value is treated internally as a delete/insert—as though the row
with the original key values was deleted and a new row with the modified key values was inserted into the table.

DELETE
This document is created with the unregistered version of CHM2PDF Pilot

The following is a positioned delete:

DELETE FROM table_name WHERE CURRENT OF cursor_name

This statement deletes from the given table the row corresponding to the current position of the cursor. This is similar
to a positioned update. Using Transact-SQL cursors, however, you can't perform positioned inserts. Since a cursor
can be thought of as a named result set, it should be clear that you insert not into a cursor but into the table (or view)
that the cursor is selecting from.

CLOSE
The CLOSE statement closes the cursor:

CLOSE cursor_name

After the cursor is closed, it can no longer be fetched from or updated/deleted. Closing the cursor deletes its keyset
while leaving the definition of the cursor intact. (That is, a closed cursor can be reopened without being redeclared.)
The keyset is regenerated during the next OPEN. Similarly, closing a static cursor deletes the temporary table of
results, which can be regenerated when the cursor is reopened. Closing a dynamic cursor doesn't have much effect
because neither the data nor the keys are materialized.

ANSI specifies that cursors are closed when a COMMIT TRAN or ROLLBACK TRAN statement is issued, but
this is not SQL Server's default behavior. In the types of applications in which cursors are most appropriate (what you
might call scrolling applications), it is common to make a change and want to both commit it and keep working within
the cursor. Having the cursor close whenever a COMMIT is issued seems inefficient and rather pointless. But if you
want to be consistent with ANSI on this, you can use the following SET option, which provides this "feature" by
closing all open cursors when a COMMIT or a ROLLBACK occurs:
SET CURSOR_CLOSE_ON_COMMIT ON

Note that when you use API server cursors with ODBC, the default is to close the cursor on COMMIT; ODBC
followed the ANSI lead here. But the SQL Server ODBC driver and the SQL Server OLE DB provider both set the
CURSOR_CLOSE_ON_COMMIT option to OFF. You can also control this behavior at the database level with
the database option close cursor on commit or by using the ALTER DATABASE command to set the
CURSOR_CLOSE_ON_COMMIT option to ON or OFF.

DEALLOCATE
DEALLOCATE, which is not part of the ANSI specification, is a cleanup command:

DEALLOCATE cursor_name

When the last cursor reference is deallocated, SQL Server releases the data structures comprising the cursor. As
you'll see shortly, with cursor variables you can have additional references to a cursor besides the original cursor name
specified in the DECLARE statement. You can deallocate one of the references, but if other references to the same
cursor exist, you can still access the cursor's rows using those other references.

Once the last reference to a cursor is deallocated, the cursor cannot be opened until a new DECLARE statement is
issued. Although it's considered standard practice to close a cursor, if you DEALLOCATE the last reference to an
open cursor it is automatically closed. The existence of DEALLOCATE is basically a performance optimization.
This document is created with the unregistered version of CHM2PDF Pilot

Without it, when would a cursor's resources be cleaned up? How would SQL Server know that you are really done
using the cursor? Probably the only alternative would be to do the cleanup anytime a CLOSE is issued. But then all
the overhead of a new DECLARE would be necessary before a cursor could be opened again. Although it is yet
another command to issue, providing a separate DEALLOCATE command lets you close and reopen a cursor
without redeclaring it.

The Simplest Cursor Syntax

In its simplest form, with no other options declared, this example cursor is forward-only (a dynamic cursor subtype):
DECLARE my_curs CURSOR FOR SELECT au_id, au_lname FROM authors

If you try any fetch operation other than FETCH NEXT (or simply FETCH, since NEXT is the default if no modifier
is specified), you get an error like this:
Msg 16911, Level 16, State 1
fetch: The fetch type LAST cannot be used with forward only cursors

Transact-SQL cursors can scroll only to the NEXT row (forward-only) unless the SCROLL option is explicitly
stated or the type of cursor is specified as STATIC, KEYSET, or DYNAMIC, in which case any fetch operation is
legal.

Fully Scrollable Transact-SQL Cursors

Here's a an example of a fully scrollable Transact-SQL cursor:
DECLARE my_curs CURSOR SCROLL FOR SELECT au_id, au_lname
FROM authors
OPEN my_curs
FETCH ABSOLUTE 6 FROM my_curs

Since the ordering of the rows in the cursor cannot be controlled or predicted without an ORDER BY, everyone
running the preceding code could get different results. Here's the output I got:
au_id au_lname
------------ --------------
427-17-2319 Dull

NOTE
To keep the focus on the central point, my examples don't check @@FETCH_STATUS or @@ERROR. Real
production-caliber code, of course, would check.

If the cursor is scrollable but not every table listed in the SELECT statement has a unique index, the cursor is created
as a static (read-only) cursor, although it is still scrollable. The results of the SELECT statement are copied into a
temporary table when the cursor is opened. Unless the FOR UPDATE OF clause is used (or the cursor is declared
with the option TYPE_WARNING), you receive no indication that this will occur. An error results after the
DECLARE cursor statement is issued:
CREATE TABLE foo
(
col1 int,
col2 int
)
This document is created with the unregistered version of CHM2PDF Pilot

-- No indexes on this table

GO

DECLARE my_curs2 SCROLL CURSOR FOR SELECT col1, col2 FROM foo
FOR UPDATE OF col1

This returns:
Server: Msg 16957, Level 16, State 4, Line 2
FOR UPDATE cannot be specified on a READ ONLY cursor.

WARNING
The error above is a compile-time error, which means that the cursor is not created at all. A subsequent attempt to
open this cursor will yield an error message that the cursor does not exist.

Some options don't make sense when used together. For example, you can't declare a cursor using FOR UPDATE
OF and combine that with either STATIC or READ_ONLY. This generates an error message when the cursor is
declared. Remember that the STATIC option and READ_ONLY are not quite equivalent. When STATIC is
specified, the results are copied into a temporary table when the cursor is opened. From then on, the results are
returned from this temporary table in the server and the updates done by other users are not reflected in the cursor
rows. A dynamic or keyset cursor can be READ_ONLY but not STATIC. As you scroll, you might see values that
have been changed by other users since the cursor was opened. A cursor declared with these characteristics doesn't
allow positioned updates or deletes via the cursor, but it doesn't prevent updates or deletes made by others from
being visible.
Beware the Dynamic Cursor
Because a dynamic cursor has no guaranteed way of keeping track of which rows are part of the cursor's result set,
you can end up with badly behaving dynamic cursors. For example, suppose you have a nonunique clustered index on
a table for which you have declared a dynamic cursor. Internally, SQL Server makes the clustered index keys unique
if necessary by adding a unique suffix to each duplicate value. If you use a positioned update to update the clustered
keys to a value that already exists, SQL Server adds the next unique suffix in sequence. (That is, the second
occurrence of any value gets a suffix of 1, the next occurrence gets a 2, and so on.) That means that the row has to
move to a place in the table after the last row with that key in order to maintain the clustered ordering sequence. If
you fetch through the cursor one row at a time and update the keys to values that already exist, you can end up
revisiting the same row, or rows, over and over. In fact, you could end up in an infinite loop.

This code does just that:

CREATE TABLE bad_cursor_table
(
a char(3),
b int IDENTITY
)
GO

CREATE CLUSTERED INDEX idx ON bad_cursor_table (a)

GO

INSERT INTO bad_cursor_table (a) VALUES ('1')

INSERT INTO bad_cursor_table (a) VALUES ('2')
INSERT INTO bad_cursor_table (a) VALUES ('3')
INSERT INTO bad_cursor_table (a) VALUES ('4')
GO

SET NOCOUNT ON
GO

DECLARE bad_cursor CURSOR DYNAMIC FOR SELECT * FROM bad_cursor_table

OPEN bad_cursor
This document is created with the unregistered version of CHM2PDF Pilot

FETCH bad_cursor
WHILE (@@fetch_status = 0)
BEGIN
UPDATE bad_cursor_table
SET a = 'new'
WHERE CURRENT OF bad_cursor
FETCH bad_cursor
END

If you execute this code, you'll see the value of column b, which you are not changing, cycle through the values 1 to 4
over and over as each update moves the updated row to the end of the table. The solution to this problem is to
always create a clustered index on a truly unique column or to not create dynamic cursors on tables without a
declared unique index.

Concurrency Control with Transact-SQL Cursors

If you use a cursor that's not in the scope of a transaction, locks are not held for long durations by default. A shared
lock is briefly requested for the fetch operation and then immediately released. If you subsequently attempt to update
through the cursor, optimistic concurrency control is used to ensure that the row being updated is still consistent with
the row you fetched into the cursor. (Locking is discussed in detail in Chapters 14 and 16.)
DECLARE my_curs SCROLL CURSOR FOR SELECT au_id, au_lname
FROM authors
OPEN my_curs
FETCH ABSOLUTE 8 FROM my_curs

-- Assume some other user modifies this row between this and
-- the next statement:
UPDATE authors SET au_lname='Kronenthal' WHERE CURRENT OF my_curs

This returns:
Server: Msg 16934, Level 10, State 1, Line 1
Optimistic concurrency check failed. The row was modified outside of
this cursor.

Server: Msg 16947, Level 10, State 1, Line 1

No rows were updated or deleted.

With optimistic concurrency control, locks are not held on the selected data, so concurrency increases. At the time of
the update, the current row in the table is compared to the row that was fetched into the cursor. If nothing has
changed, the update occurs. If a change is detected, the update does not occur and error message 16934 is returned.
The term "optimistic" is meant to convey that the update is attempted with the hope that nothing has changed. A check
is made at the time of the update to ensure that, in fact, nothing has changed. (It's not so optimistic that it forgoes the
check.) In this case, although a change is detected and the update is refused, no locks are issued to prevent such a
change. You could decide to go ahead and update the row even though it failed the optimistic concurrency check. To
do so, you would fully qualify the update with a normal WHERE clause specifying the unique key instead of issuing a
positioned update (WHERE CURRENT OF my_curs). For example:
UPDATE authors
SET au_lname='Kronenthal'
WHERE au_id='427-17-2319'

Although here we are looking at this situation via the Transact-SQL syntax, in the real world you're much more likely
to encounter this situation using API server cursors. In such a case, you might use optimistic concurrency control. If a
This document is created with the unregistered version of CHM2PDF Pilot

conflict is encountered, you can display a message to your end user—something like "The row has subsequently
changed. Here is the new value [which you could refetch to get]. Do you still want to go ahead with your update?"

There are actually two types of optimistic concurrency. With Transact-SQL cursors, if the table has a rowversion
(or timestamp) column defined, a cursor update uses it to determine whether the row has been changed since it was
fetched. (Recall that a rowversion column is an automatically increasing value that is updated for any change to the
row. It is unique within the database, but it bears no relationship to the system or calendar's date and time.) When the
update is attempted, the current rowversion on the row is compared to the rowversion that was present when the
row was fetched. The rowversion is automatically and invisibly fetched into the cursor even though it was not declared
in the select list. Be aware that an update will fail the optimistic concurrency check even if a column not in the cursor
has been updated since the row was fetched. The update to that column (not in the cursor) will increase the
rowversion, making the comparison fail. But this happens only if a rowversion column exists on the table being
updated.

If there is no rowversion column, optimistic concurrency control silently reverts to a "by-value" mode. Since no
rowversion exists to indicate whether the row has been updated, the values for all columns in the row are compared
to the current values in the row at the time of the update. If the values are identical, the update is allowed. Notice that
an attempt to update a column in a row that is not part of the cursor causes optimistic concurrency control to fail,
whether it is rowversion based or value based. You can also use traditional locking in conjunction with a cursor to
prevent other users from changing the data that you fetch.

Although optimistic concurrency is the default for updateable cursors, you can designate this kind of locking explicitly
by using the keyword OPTIMISTIC in your DECLARE statement.

Optimistic concurrency control works quite well in environments with relatively low update contention. With high
update activity, optimistic concurrency control might not be appropriate; for example, you might not be able to get an
update to succeed because the row is being modified by some other user. Whether optimistic or pessimistic (locking)
concurrency control is more appropriate depends on the nature of your application. An address book application with
few updates is probably well suited to optimistic concurrency control. An application used to sell a product and keep
track of inventory is probably not.

As another example, if you write an application to sell reserved tickets for a flight and sales are brisk, optimistic
concurrency control is probably not appropriate. Here's what the telephone conversation between the ticket agent
and the customer might sound like if you use optimistic concurrency control:

Caller: "I'd like two tickets to Honolulu for January 15."

Ticket agent: "OK, I have two seats on flight number 1 leaving at 10 a.m."

Caller: "Great, I'll take them."

Ticket agent: "Oops. I just tried to reserve them, and they've been sold by another agent. Nothing else seems
to be available on that date."

If you want to make sure that no other users have changed the data that you've fetched, you can force SQL Server
to hold update locks on data that is fetched while the cursor is open or until the next fetch. You can do this by
specifying the SCROLL_LOCKS option in the cursor's DECLARE statement. These update locks prevent other
users from updating the cursor's row or reading the row through another cursor using SCROLL_LOCKS, but they
do not prevent other users from reading the data. Update locks are held until the next row is fetched or until the
cursor is closed, whichever occurs first. If you need full pessimistic currency control, you should read the cursor row
inside a transaction; these locks are held until the transaction is either committed or rolled back.

Here's an example of a cursor that holds scroll locks while the cursor is open, ensuring that another user cannot
update the fetched row:
This document is created with the unregistered version of CHM2PDF Pilot

DECLARE my_curs CURSOR SCROLL SCROLL_LOCKS FOR SELECT au_id, au_lname

FROM authors
OPEN my_curs
FETCH ABSOLUTE 12 FROM my_curs
-- Pause to create window for other connection.
-- From other connection, try to update the row. E.g.,
-- UPDATE authors SET au_lname='Newest val' WHERE au_id='486-29-1786'.
-- The update will be blocked by a KEY UPDATE lock until this cursor is
-- closed. Refetch the row next and see that it has not changed.
-- The CLOSE cursor will release the SH_PAGE lock.

FETCH RELATIVE 0 FROM my_curs

-- RELATIVE 0 = "freshen" current row
CLOSE my_curs
DEALLOCATE my_curs

In this example, we're only reading data, not modifying it. When you do positioned updates or deletes, you might
want to group all the cursor operations as a single transaction. To do that, you use an explicit BEGIN TRAN
statement. The fetch requests an update (scroll) lock, and the lock is retained until the end of the transaction.
NOTE
Don't get confused between update and scroll locks. Scroll locks for cursor processing are a special type of update
lock. Both scroll and update locks prevent other processes from acquiring exclusive locks, update locks, or scroll
locks, but they don't prevent simple share (read) locks. The real difference between update locks and scroll locks is
the length of time the lock is held. If there is no surrounding transaction, scroll locks are released when a new row is
fetched or the cursor is closed. If there is a surrounding transaction, the scroll lock is held until the end of the
transaction. (This is a slight oversimplification. We'll discuss the difference between scroll and update locks in more
detail in Chapter 14.)

The following example shows optimistic concurrency control within a transaction. This means we do not want to
acquire scroll locks. A simple share lock on the row is acquired while the row is being read, but it is released as soon
as the row is read. If you use another connection to observe the locks, you'll see that the shared row locks are not
held, although of course the exclusive locks, which are issued to carry out the actual update, are held until the
transaction completes.
-- Be sure the default setting is in effect
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
DECLARE my_curs CURSOR SCROLL FOR SELECT au_id, au_lname
FROM authors
OPEN my_curs
BEGIN TRAN

FETCH FIRST FROM my_curs

IF (@@FETCH_STATUS <> 0) -- If not a valid fetch, get out
PRINT 'Fetch is not valid'
-- Verify from another connection
-- that no lock held using sp_lock

UPDATE authors SET au_lname='Row1 Name' WHERE CURRENT OF my_curs

IF (@@ERROR <> 0)
PRINT 'Update not valid'

FETCH LAST FROM my_curs

IF (@@FETCH_STATUS <> 0) -- If not a valid fetch, discontinue

PRINT 'Fetch not valid'

-- Verify from another connection

-- and see the exclusive KEY lock from previous update
This document is created with the unregistered version of CHM2PDF Pilot

UPDATE authors SET au_lname='LastRow Name' WHERE CURRENT OF my_curs

IF (@@ERROR <> 0)
PRINT 'Update not valid'
ROLLBACK TRANCLOSE my_curs
DEALLOCATE my_curs

The previous example uses a transaction, but it still uses optimistic concurrency control. If you want to be sure that a
row doesn't change once you fetch it, you must hold the shared locks. You can do this by using the HOLDLOCK
hint in the SELECT statement or by setting the isolation level to Repeatable Read (or Serializable), as in this example:
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
DECLARE my_curs CURSOR SCROLL FOR SELECT au_id, au_lname
FROM authors
OPEN my_curs
BEGIN TRAN

FETCH FIRST FROM my_curs

IF (@@FETCH_STATUS <> 0) -- If not a valid fetch, discontinue
PRINT 'Fetch not valid'
-- Check locks from another connection using sp_lock

UPDATE authors SET au_lname='Newer Row1 Name'

WHERE CURRENT OF my_curs

IF (@@ERROR <> 0)
PRINT 'Update not valid'

FETCH LAST FROM my_curs

IF (@@FETCH_STATUS <> 0) -- If not a valid fetch, discontinue

PRINT 'Fetch not valid'

-- Verify from another connection

-- and see the exclusive KEY lock from previous update, and
-- shared KEY from most recent FETCH

UPDATE authors SET au_lname='Newer LastRow Name'

WHERE CURRENT OF my_curs
IF (@@ERROR <> 0)
PRINT 'Update not valid'

ROLLBACK TRAN

CLOSE my_curs
DEALLOCATE my_curs

This example holds onto shared locks on all the rows that have been fetched. You can be sure that once a row is
fetched, another user cannot change it while the transaction is in process. However, you cannot be sure that you can
update the row. If you take this example and run it simultaneously in two SQL Query Analyzer windows, you might
get the following error in one of the two windows:
Msg 1205, Level 13, State 1
Your server command (process id 14) was deadlocked with another
process and has been chosen as deadlock victim. Re-run your command

To increase the chance of deadlock (for illustrative purposes only—you'd never really want to increase your chances
of deadlock) you can introduce a delay before each of the UPDATE statements:
WAITFOR DELAY '00:00:10'
This document is created with the unregistered version of CHM2PDF Pilot

This is a classic conversion deadlock, which occurs when each process holds a shared lock on a row and each
needs an exclusive lock on the same row. Neither can get the exclusive lock because of the other process's shared
lock. No matter how long the two processes wait, no resolution occurs because neither can get the lock it needs.
SQL Server terminates one of the processes so that the other can conclude. (Chapter 14 describes such deadlocks in
more detail. It also differentiates between ROW and KEY locks, which we're treating as the same thing in this
discussion.) To avoid conversion deadlocks, you should use scroll locks instead of shared locks for any data you are
reading if you intend to update it later. A scroll lock or an update lock does not prevent another process from reading
the data, but it ensures that the holder of the update lock is next in line for an exclusive lock for that resource and thus
eliminates the conversion deadlock problem by serializing access for the update lock resource.

The following code is a modification of the previous example that uses the SCROLL_LOCKS keyword in the cursor
declaration. This forces SQL Server to acquire an update lock on the row being fetched, which is held until the next
row is fetched. No other process can acquire another update lock on this same row. Since we're updating the fetched
row, the update lock is converted into an exclusive lock, which is held until the end of the transaction. If we run 2 (or
even 200) instances of this simultaneously, we do not get a deadlock. The scroll (update) lock acquired the first time
the row is fetched makes the multiple instances run serially, since they must queue and wait for the update lock to be
released:
-- We will use SCROLL_LOCKS, which are held until the
-- next row is fetched or until they convert into EXCLUSIVE locks.
-- So we do not require REPEATABLE READ. We can use the default
-- isolation level of READ COMMITTED.
SET TRANSACTION ISOLATION LEVEL READ COMMITTED

DECLARE my_curs CURSOR SCROLL SCROLL_LOCKS FOR SELECT au_id, au_lname

FROM authors
OPEN my_curs
BEGIN TRAN

FETCH FIRST FROM my_curs

IF (@@FETCH_STATUS <> 0) -- If not a valid fetch, get out
GOTO OnError
WAITFOR DELAY '00:00:10'

UPDATE authors SET au_lname='Newer Row1 Name'

WHERE CURRENT OF my_curs

IF (@@ERROR <> 0)
GOTO OnError

FETCH LAST FROM my_curs

IF (@@FETCH_STATUS <> 0) -- If not a valid fetch, get out

GOTO OnError

-- Delay for 10 secs to increase chances of deadlocks

WAITFOR DELAY '00:00:10'

UPDATE authors SET au_lname='Newer LastRow Name'

WHERE CURRENT OF my_curs
IF (@@ERROR <> 0)
GOTO OnError

COMMIT TRAN
IF (@@ERROR=0)
PRINT 'Committed Transaction'
GOTO Done

OnError:
PRINT 'Rolling Back Transaction'
This document is created with the unregistered version of CHM2PDF Pilot

ROLLBACK TRAN

Done:
CLOSE my_curs
DEALLOCATE my_curs

NOTE
If we didn't do an explicit BEGIN TRAN in the example above, we would still get an UPDATE lock on the most
recently fetched row. This lock would then move to subsequent rows and be released from the earlier ones as we
continue to fetch.

Optimistic Concurrency Control vs. Locking

Whether you should use optimistic concurrency control, use scroll locks, or hold shared locks depends on the degree
to which you experience update conflicts or deadlocks. If the system is lightly updated or the updates tend to be quite
dispersed and few conflicts occur, optimistic concurrency control will offer the best concurrency since locks are not
held. But some updates will be rejected, and you have to decide how to deal with that. If updates are frequently
rejected, you should probably move to the more pessimistic and traditional mode of locking. Holding shared locks
prevents rows from being changed by someone else, but it can lead to deadlocks. If you are getting frequent
deadlocks, you should probably use scroll locks.

Note that the FOR UPDATE OF modifier of the DECLARE statement does not affect locking behavior, and it does
not mean that scroll (update) locks are used. The modifier's only purpose is to allow you to ensure that the cursor is
updateable and that the user has the necessary permissions to make updates. Normal optimistic concurrency is used if
SCROLL_LOCKS is not specified, which means that a shared lock is acquired on the row being read and is
released as soon as the read is completed. If a cursor cannot be updated, you get an error message when a cursor is
declared with FOR UPDATE OF. You could argue that FOR UPDATE OF should use update locks rather than
shared locks on the fetched data, although there is certainly a case to be made for not doing it this way. Even though
FOR UPDATE OF is specified, an update might never be made. If no update occurs, using shared locks is best
because someone else can still declare the same cursor and there is no need to serialize the running of that batch.
Concurrency therefore increases.

You should also realize that you do not have to issue the FOR UPDATE OF modifier to be able to do positioned
updates via the cursor. You can update a cursor that does not have the FOR UPDATE OF modifier specified as long
as the underlying declared cursor is updateable. A SELECT statement cannot be updated if it uses GROUP BY,
HAVING, UNION, or CASE. But you do not get an error message stating that the SELECT statement cannot be
updated until you actually try to do an update. By specifying FOR UPDATE OF in the DECLARE statement, you
find out right away if the SELECT cannot be updated. You should consider always specifying FOR UPDATE OF for
any cursor that you might later update.

If you will usually update fetched rows and there is contention for updates, you should use the SCROLL_LOCKS
keyword. Yes, this serializes access to that resource, but the alternative is a high incidence of rejected updates with
optimistic concurrency control or deadlocks with holding shared locks. Both of these require a lot of retries, which is
wasteful. Using scroll locks in this type of scenario probably allows you to write the simplest application and gives you
the best throughput for your system.

A final comment regarding concurrency options: if you use FOR UPDATE OF, you can update any column in the
cursor. (You might be able to update any column if you don't specify the option, but you won't know that until you try
the positioned update.) But if you use the FOR UPDATE OF column_list syntax and then try to update a column
that is not in your specified list, you get error number 16932.

A lot of the discussion of cursor concurrency options ties in to modes and types of locking in SQL Server. You might
consider rereading this section after you absorb all the details about locking in Chapter 14.
This document is created with the unregistered version of CHM2PDF Pilot

34

Cursor Variables
SQL Server 2000 lets you declare variables with a type of cursor. To assign a cursor to a cursor variable, you use
the SET statement. Cursors variables cannot be assigned a value with an assignment SELECT, as scalar values can.
Cursor variables can be either the source or the target in SET assignment statements, and they can be used as the
type for OUTPUT parameters. They can be used in any of the cursor management statements: OPEN, FETCH,
CLOSE, and DEALLOCATE. However, you cannot have a column in a table of type cursor, and input parameters
to stored procedures cannot be cursors.

The following example declares a cursor and then sets a variable to reference the same result set:
DECLARE declared_cursor CURSOR FOR SELECT au_lname FROM authors
DECLARE @cursor_var CURSOR
SET @cursor_var = declared_cursor

The declared cursor and the cursor variable are almost synonymous. You can use either the declared name or the
cursor variable name to open the cursor—they are two references to the same internal structure. You can open the
cursor with this statement:
OPEN @cursor_var

Once you do this, the cursor is open. If you then try to open it with this declared name
OPEN declared_cursor

you get this error:

Msg 16905, Level 16, State 1
The cursor is already open.

If you issue one FETCH with the declared cursor, a subsequent FETCH from the cursor variable retrieves the next
row. A CLOSE using either reference closes the cursor, and no more rows can be fetched. The only time the two
references are treated differently is in the DEALLOCATE statement. Deallocating one of the references means only
that you can no longer access the cursor using that reference. The internal data structures are not freed until the last
reference is deallocated. If the cursor has not yet been closed, only the deallocation of the last reference closes it.

You can have more than one cursor variable referencing the same cursor, as the following example shows:
DECLARE declared_cursor CURSOR LOCAL
FOR SELECT title_id, price FROM titles
DECLARE @cursor_var_a CURSOR
DECLARE @cursor_var_b CURSOR
SET @cursor_var_a = declared_cursor
SET @cursor_var_b = @cursor_var_a
OPEN @cursor_var_a -- opens declared cursor
FETCH @cursor_var_b -- fetches row 1 from declared_cursor
FETCH declared_cursor -- fetches row 2
DEALLOCATE declared_cursor
-- keeps cursor open since other references remain,
-- but can't use declared_cursor name to reference the cursor
This document is created with the unregistered version of CHM2PDF Pilot

One main use of cursor variables is for output parameters. There are four stored procedures for retrieving information
about cursors, and each returns its information in a cursor. We'll look at these procedures in the next section.

Obtaining Cursor Information

Along with the wealth of cursor functionality in SQL Server 2000, you have access to a function and several
procedures that provide you with information about your cursors. However, these procedures are not always easy to
use, so I've provided some additional procedures that go beyond what is offered with the installed SQL Server
product.

CURSOR_STATUS
The CURSOR_STATUS function determines the state of a particular cursor or cursor variable. Five possible values
can be returned from this function, and the meaning of the values is slightly different depending on whether the
argument is a cursor name or a cursor variable. The function takes two arguments; the first one indicates a global
cursor, a local cursor, or a cursor variable (which is always local). Here's the syntax:
CURSOR_STATUS
(
{'local', 'cursor_name'}
| {'global', 'cursor_name'}
| {'variable', 'cursor_variable'}
)

Table 13-3 shows the five possible return values and their meanings.

Table 13-3. Meanings of return values from the CURSOR_STATUS function.

Return Value Cursor Name Cursor Variable
The cursor is open, and for static and The cursor allocated to this variable is
keyset cursors the result set has at open, and for static and keyset
1 least one row; for dynamic cursors, cursors the result set has at least one
the result set can have zero, one, or row, for dynamic cursors, the result
more rows. set can have zero, one or more rows.
The cursor allocated to this variable is
The result set of the cursor is empty.
open, but the result set is empty.
0 (Dynamic cursors never return this
(Dynamic cursors never return this
result.)
result.)
The cursor allocated to this variable is
-1 The cursor is closed.
closed.
This document is created with the unregistered version of CHM2PDF Pilot

No cursor was assigned to this

OUTPUT variable by the previously
called procedure.

OR
A cursor was assigned by the
previously called procedure, but it
-2 Not applicable. was in a closed state when the
procedure completed. Therefore, the
cursor is deallocated and not returned
to the calling procedure.

OR
There is no cursor assigned to a
declared cursor variable.
A cursor variable with the specified
name does not exist.
A cursor with the specified name does
-3
not exist. OR
If one exists it has not yet had a
cursor allocated to it.

We'll use the CURSOR_STATUS function below when we check the output parameter from the
sp_describe_cursor stored procedure.

sp_describe_cursor
This procedure provides information on one cursor or cursor variable whose name must be passed to the procedure.
Even though only one row of information is returned, you must supply the procedure with an output parameter of
type cursor. After executing the procedure, you must use the cursor FETCH command to retrieve the information
from the returned cursor. Here's the syntax for calling the stored procedure:

sp_describe_cursor [@cursor_return =] output_cursor_variable OUTPUT

{
[, [@cursor_source =] N'local',
[@cursor_identity =] N'local_cursor_name'] |

[, [@cursor_source =] N'global',
[@cursor_identity =] N'global_cursor_name'] |

[, [@cursor_source =] N'variable',
[@cursor_identity =] N'input_cursor_variable']
}

Here's an example of using the procedure and then accessing the data returned in the cursor variable.
Authors_cursor is the cursor for which we want information and @Report is a cursor variable to hold the results.
DECLARE authors_cursor CURSOR KEYSET FOR
SELECT au_lname
FROM authors
WHERE au_lname LIKE 'S%'

OPEN authors_cursor

-- Declare a cursor variable to hold the cursor output variable

-- from sp_describe_cursor
This document is created with the unregistered version of CHM2PDF Pilot

DECLARE @Report CURSOR

-- Execute sp_describe_cursor into the cursor variable

EXEC master.dbo.sp_describe_cursor
@cursor_return = @Report OUTPUT,
@cursor_source = 'global',
@cursor_identity = 'authors_cursor'

-- Verify that the cursor variable contains information

IF cursor_status('variable', '@Report') != 1
PRINT 'No information available from the cursor'

ELSE BEGIN
-- Fetch all the rows from the sp_describe_cursor output cursor
WHILE (@@fetch_status = 0)
BEGIN
FETCH NEXT FROM @Report
END
END

-- Close and deallocate the cursor from sp_describe_cursor

IF cursor_status('variable', '@Report') >= -1
BEGIN
CLOSE @Report
DEALLOCATE @Report
END
GO

-- Close and deallocate the original cursor

CLOSE authors_cursor
DEALLOCATE authors_cursor
GO

Here's some of the information returned:

reference_name cursor_scope status model concurrency scrollable
-------------- ------------ ------ ----- ----------- ----------
authors_cursor 2 1 2 3 1

open_status cursor_rows fetch_status column_count row_count

----------- ----------- ------------ ------------ ---------
1 3 -9 1 0

last_operation
--------------
1

Table 13-4 explains some of the return values.

Table 13-4. Meaning of values returned by sp_describe_cursor.

Result Column Possible Return Values
1 = LOCAL
CURSOR_SCOPE
2 = GLOBAL
The same values reported by the CURSOR_STATUS
STATUS
system function, described above.
This document is created with the unregistered version of CHM2PDF Pilot

1 = Static

2 = Keyset
MODEL
3 = Dynamic

4 = Fast forward
1 = Read-only

CONCURRENCY 2 = Scroll locks

3 = Optimistic
0 = Forward-only
SCROLLABLE
1 = Scrollable
0 = Closed
OPEN_STATUS
1 = Open
CURSOR_ROWS Number of qualifying rows in the result set.
The status of the last fetch on this cursor, as indicated by
the @@FETCH_STATUS function.

0 = The fetch was successful.

FETCH_STATUS -1 = The fetch failed or is beyond the bounds of the

cursor.

-2 = The requested row is missing.

-9 = There was no fetch on the cursor.

COLUMN_COUNT The number of columns in the cursor result set.
The number of rows affected by the last operation on the
ROW_COUNT
cursor, as indicated by the @@ROWCOUNT function.
The last operation performed on the cursor.

0 = No operations have been performed on the cursor.

1 = OPEN

2 = FETCH

LAST_OPERATION 3 = INSERT

4 = UPDATE

5 = DELETE

6 = CLOSE

7 = DEALLOCATE

sp_cursor_list
This document is created with the unregistered version of CHM2PDF Pilot

This procedure returns the same information as sp_describe_cursor, but it does so for all cursors. You can specify
whether you want all global cursors, all local cursors, or both. Here's the syntax:

sp_cursor_list [@cursor_return =] cursor_variable_name OUTPUT,

[@cursor_scope =] cursor_scope

You can actually create your own stored procedure as a wrapper around sp_cursor_list. This allows you to avoid
the declaration of the @Report cursor output parameter every time you want to look at cursor properties. You also
do not have to code the repeated fetching of rows using this cursor variable. Below is an example of what such a
procedure might look like. It uses a default of 3 for the value of scope, which means all cursors, but you can override
that when you call this new sp_cursor_info procedure. You can pass a 1 to indicate local cursors only or a 2 to
indicate global cursors only:
USE master
GO

CREATE PROC sp_cursor_info

(@scope int = 3)
AS
-- Declare a cursor variable to hold the cursor output variable
-- from sp_cursor_list.

DECLARE @Report CURSOR

-- Execute sp_cursor_list into the cursor variable.

EXEC master.dbo.sp_cursor_list @cursor_return = @Report OUTPUT,

@cursor_scope = @scope

-- Fetch all the rows from the sp_cursor_list output cursor.

FETCH NEXT FROM @Report
WHILE (@@FETCH_STATUS = 0)
BEGIN
FETCH NEXT FROM @Report
END

-- Close and deallocate the cursor from sp_cursor_list.

CLOSE @Report
DEALLOCATE @Report
RETURN
GO

The procedure returns individual cursor rows, one for each cursor available in the session. You can expand this
procedure even further to populate a temporary table with the values in the returned row. You can then use the CASE
functionality to translate the integer code numbers returned into meaningful strings. The companion CD includes a
script to build such a procedure, called sp_cursor_details, which calls sp_cursor_list and receives the results into a
cursor variable. By creating the sp_cursor_details procedure in your master database, you can execute
sp_cursor_details just like any other system-supplied stored procedure. I'll declare the same authors_cursor I
declared earlier, and then execute the sp_cursor_details procedure. The output is split into three sets of columns so it
all fits on the page of this book, but the output in Query Analyzer would be all on one line.
DECLARE authors_cursor CURSOR KEYSET FOR
SELECT au_lname
FROM authors
WHERE au_lname LIKE 'S%'
GO

OPEN authors_cursor
This document is created with the unregistered version of CHM2PDF Pilot

EXEC sp_cursor_details

OUTPUT:
Cursor Reference Declared Name Scope
------------------------------ ------------------------------ --------
authors_cursor authors_cursor global

Status Model LOCKING Scrolling Open Qualifying Rows

------ ------------ ------------ ------------ ---- ---------------
open keyset optimistic scrollable yes 3

Fetch Status Columns Rows for last operation Last operation Handle
------------- ------- ----------------------- -------------- ---------
no fetch done 1 0 OPEN 180150000

sp_describe_cursor_tables
This procedure reports on the tables accessed by a given cursor. It returns its output in a cursor variable, with one
row for each table in the specified cursor. It returns such information as the name of the table and any optimizer hints
specified for the table. See the SQL Server online documentation for more details.

sp_describe_cursor_columns
This procedure reports the attributes of the columns in the result set of a given cursor. It returns its output in a cursor
variable, with one row for each column in the SELECT list of the specified cursor. The information it returns includes
the name of the column, the datatype, the ordinal position of the column, and the ID of the table from which the cursor
is selected. See the SQL Server online documentation for more details.
This document is created with the unregistered version of CHM2PDF Pilot

34

Working with Text and Image

Data
SQL Server provides large object (LOB) support via the ntext, text, and image datatypes. I talked about their
storage structure in Chapter 6. If you work with these datatypes, you might want to use the additional statements
provided by SQL Server along with the standard SELECT, INSERT, UPDATE, and DELETE statements. Because
a single LOB column can be as large as 2 GB, you frequently need to work with LOB data in chunks, and these
additional statements (which we'll see in a moment) can help. I could have discussed this topic earlier, when I
discussed Transact-SQL programming in Chapter 10. However, because you need some knowledge of isolation
levels, transactions, and consistency issues to understand this topic, I've waited until after I covered most of those
issues. In addition, one of the examples I'll show you involves cursors, so I really couldn't show that one before now.

For simplicity's sake, most of the discussion will deal with the text datatype. But everything here is also relevant to
the image and ntext datatypes. These three datatypes are essentially the same internally. Recall that ntext, text, and
image datatypes are special because they are normally not stored on the same data page as the rest of the row.
Instead, the data row holds only a 16-byte pointer to an ntext, text, or image structure. Remember that you can
choose to store LOB data in the row itself if the amount of data is small enough and the table option text in row has
been enabled.

Although the space required by text data is managed much more efficiently in SQL Server 2000 than in earlier
versions, functional drawbacks still exist. Although you can indeed use standard INSERT, UPDATE, DELETE, and
SELECT statements with a text column, some significant restrictions apply. In a WHERE clause, you can search on
the text column only with the LIKE operator or with the functions SUBSTRING and PATINDEX. Variables of type
text cannot be declared or manipulated. You can declare a parameter in a stored procedure to be of type text (or
ntext or image), but you can't do much besides pass a value to the procedure initially. For example, you cannot
subsequently assign different values to the parameter or return it as an OUTPUT parameter. Because of these
limitations, you'll want to use these special datatypes only when another datatype isn't a reasonable option. If a
varchar(8000) column can work for you, (or nvarchar(4000) for Unicode data) you can use it and avoid text or
ntext altogether. But if you absolutely need a memo field, for example, and 8000 characters are not enough, you'll
need to use text or denormalize the data and use multiple varchar columns.

If a text column makes the most sense for you despite its drawbacks, you need to understand how to work
effectively with text. When you can, it is easiest to work with text datatypes using standard SELECT, INSERT,
UPDATE, and DELETE statements. But if your text data gets large, you'll run into issues (such as how big a string
your application can pass) that might make it necessary for you to deal with chunks of data at a time instead of the
entire column.

The special statements for working with text data are WRITETEXT, READTEXT, and UPDATETEXT. Both
READTEXT and UPDATETEXT let you work with chunks of a text column at a time. The WRITETEXT statement
does not let you deal with chunks but rather with the entire column only.

You'll need to consider the overhead of transaction logging done with these operations. Unlike the normal INSERT,
UPDATE, and DELETE operations that always cause the entire row to be logged, the amount of logging done by the
WRITETEXT and UPDATETEXT operations depends on the recovery model you're using. These two operations
are considered BULK operations, as I discussed in Chapter 5, which means that if you're in FULL or SIMPLE
recovery mode, all the LOB data will be written to the log. If you're in BULK LOGGED mode, the LOB data isn't
written directly to the log; instead, all pages modified by these BULK operations will be backed up when the
transaction log is backed up. Also, if you use SQL Server replication to replicate text or image columns, any
This document is created with the unregistered version of CHM2PDF Pilot

UPDATETEXT or WRITETEXT operations must be logged because the replication process looks for changes
based on the transaction log.

The WRITETEXT, READTEXT, and UPDATETEXT statements all work with a text pointer. A text pointer is a
unique varbinary(16) value for each text, ntext, or image column of each row. The following discussion of the three
operations will talk about text data for simplicity, but in most cases it will apply to all three of the LOB datatypes.

WRITETEXT
WRITETEXT completely overwrites an existing text column. You provide the column name (qualified by the table
name), the text pointer for the specific column of a specific row, and the actual data to be written. It might seem like a
catch-22 when using WRITETEXT initially, because you need to pass it a text pointer—but if the column is initially
NULL, there is no text pointer. So how do you get one? You SELECT it using the textptr function. But if the text
column has not been initialized, the textptr function returns NULL. To initialize a text pointer for a column of a row
with text or image data, you can perform some variation of the following:
 Explicitly insert a non-null value in the text column when you use an INSERT statement. Recognize that
WRITETEXT will completely overwrite the column anyway, so the value can always be something such as A
or a space.
 Define a default on the column with a non-null value such as A. Then when you do the insert, you can specify
DEFAULT or omit the column, which will result in the default value being inserted and the text pointer being
initialized.
 Update the row after inserting it, explicitly setting it to anything, even NULL.

You then select the text pointer into a variable declared as varbinary(16) and pass that to WRITETEXT. You can't
use SELECT statements or expressions in the WRITETEXT statement. This means that the statement is limited to
being executed one row at a time, although it can be done from within a cursor. The SELECT statement that gets the
text pointer should be known to return only one row, preferably by using an exact match on the primary key value in
the WHERE clause, because that will ensure that at most one row can meet the criteria. You can, of course, use
@@ROWCOUNT to check this if you're not absolutely sure that the SELECT statement can return only one row.
Before using the WRITETEXT statement, you should also ensure that you have a valid text pointer. If you find a row
with the criteria you specified and the text pointer for that row was initialized, it will be valid. You can check it as a
separate statement using the TEXTVALID function. Or you can check that you do not have a NULL value in your
variable that was assigned the text pointer, as you'll see in the following example. Make sure that you don't have an
old text pointer value from a previous use, which would make the IS NOT NULL check be TRUE. In this example, I
do one variable assignment and the variable starts out NULL, so we can be sure that a non-null value means we have
selected a valid text pointer:
-- WRITETEXT with an unprotected text pointer
DECLARE @mytextptr varbinary(16)
SELECT @mytextptr=TEXTPTR(pr_info)
FROM pub_info WHERE pub_id='9999'
IF @mytextptr IS NOT NULL
WRITETEXT pub_info.pr_info @mytextptr 'Hello Again'

In this example, the text pointer is not protected from changes made by others. Therefore, it is possible that the text
pointer will no longer be valid by the time the WRITETEXT operation is performed. Suppose that you get a text
pointer for the row with pub_id='9999'. But before you use it with WRITETEXT, another user deletes and reinserts
the row for publisher 9999. In that case, the text pointer you're holding will no longer be valid. In the example above,
the window for this occurrence is small, since we do the WRITETEXT immediately after getting the text pointer. But
there is still a window of vulnerability. In your application, the window might be wider. If the text pointer is not
valid when you do the WRITETEXT operation, you will get an error message like this:
Msg 7123, Level 16, State 1
Invalid text pointer value 000000000253f380.
This document is created with the unregistered version of CHM2PDF Pilot

You can easily see this for yourself if you add a delay (for example, WAITFOR DELAY '00:00:15') after getting the
text pointer and then delete the row from another connection. You'll get error 7123 when the WRITETEXT operation
executes. If you think the chances of getting this error are slim, you can choose to simply deal with the error if it
occurs.

You should instead consider using transaction protection to ensure that the text pointer will not change from the time
you read it until you use it, and to serialize access for updates so that you do not encounter frequent deadlocks. Many
applications use TEXTVALID to check right before operating—that's the right idea, but it's hardly foolproof. There's
still a window between the TEXTVALID operation and the use of the text pointer during which the text pointer can
be invalidated. The only way to close the window is to make both operations part of an atomic operation. This means
using a transaction and having SQL Server protect the transaction with a lock. (For more about locking, see Chapter
14.)

By default, SQL Server operates with Read Committed isolation and releases a share (READ) lock after the page
has been read. So simply putting the pointer in a transaction with the Read Committed isolation level, which is SQL
Server's default, is not enough. You need to ensure that the lock is held until the text pointer is used. You could
change the isolation level to Repeatable Read, which is not a bad solution, but this changes the isolation behavior for
all operations on that connection, so it might have a more widespread effect than you intend. (Although you could, of
course, then change it right back.) Also, this approach doesn't guarantee that you will subsequently be able to get the
exclusive lock required to do the WRITETEXT operation; it ensures only that when you get to the WRITETEXT
operation, the text pointer will still be valid. You won't be sure that you're not in the lock queue behind another
connection waiting to update the same row and column. In that case, your transaction and the competing one would
both hold a share lock on the same row and would both need to acquire an exclusive lock. Since both transactions
are holding a share lock, neither can get the exclusive lock and a deadlock results in one of the connections having its
transaction automatically aborted. (In Chapter 14, you'll see that this is an example of a conversion deadlock.) If
multiple processes are intending to modify the text and all transactions first request an update lock in a transaction
when selecting the text pointer, conversion deadlocks will be avoided because only one process will get the update
lock and the others will queue for it. But users' transactions that need only to read the row will not be affected
because an update lock and a share lock are compatible.

Using the update lock on the text pointer is good for serializing access to the actual text data, even though the lock on
the text data is distinct from the lock on the row with the text pointer. In this case, you essentially use the update lock
on a text pointer as you'd use an intent lock for the text data. This is conceptually similar to the intent locks that SQL
Server uses on a table when a page-locking or row-locking operation for that table will take place. That operation
recognizes that rows, pages, and tables have an implicit hierarchy. You can think of text pointers and text data as
having a similar hierarchy and use the update lock on the text pointer to protect access to the associated text data.

Here is the improved version that protects the text pointer from getting invalidated and also reserves the transaction's
spot in the queue so that it will get the exclusive lock that's necessary to change the column. This approach will avoid
conversion deadlocks on the text pages:
-- WRITETEXT with a properly protected text pointer
BEGIN TRAN
DECLARE @mytextptr varbinary(16)
SELECT @mytextptr=TEXTPTR(pr_info)
FROM pub_info (UPDLOCK) WHERE pub_id='9999'
IF @mytextptr IS NOT NULL
WRITETEXT pub_info.pr_info @mytextptr 'Hello Again'
COMMIT TRAN

READTEXT
READTEXT is used in a similar way to WRITETEXT, except that READTEXT allows you to specify a starting
This document is created with the unregistered version of CHM2PDF Pilot

position and the number of bytes to read. This is its basic syntax:

READTEXT [[database.]owner.]table_name.column_name
text_ptr offset size [HOLDLOCK]

Unlike with WRITETEXT, with READTEXT you do not need to work with the entire contents of the data. You can
specify the starting position (offset) and the number of bytes to read (size). READTEXT is often used with the
PATINDEX function to find the offset (in number of characters) at which some string or pattern exists, and it's also
used with datalength to determine the total size of the text column in bytes. If you're working with ntext data, you'll
have to divide DATALENGTH by 2 to get the number of characters. But you cannot directly use these functions as
the offset parameter. Instead, you must execute them beforehand and keep their values in a local variable, which you
then pass. As I mentioned in the discussion of WRITETEXT, you'll want to protect your text pointer from becoming
invalidated. In the next example, you'll read text without updating it. So you can use the HOLDLOCK lock hint on
the SELECT statement for the text pointer (or set the isolation level to Repeatable Read).

Sometimes people think that transactions are used only for data modifications, but notice that in this case you use a
transaction to ensure read repeatability (of the text pointer) even though you're not updating anything. You can
optionally add HOLDLOCK to the READTEXT statement to ensure that the text doesn't change until the transaction
has completed. But in the example below, we'll read the entire contents with just one read and we will not be
rereading the contents, so there's no point in using HOLDLOCK. This example finds the pattern Washington in the
pr_info column for pub_id 0877 and returns the contents of that column from that point on:
-- READTEXT with a protected text pointer
BEGIN TRAN
DECLARE @mytextptr varbinary(16), @sizeneeded int, @pat_offset int
SELECT @mytextptr=TEXTPTR(pr_info),
@pat_offset=PATINDEX('%Washington%',pr_info) - 1,
@sizeneeded=DATALENGTH(pr_info) -
(PATINDEX('%Washington%',pr_info) - 1)
FROM pub_info (HOLDLOCK) WHERE pub_id='0877'

IF @mytextptr IS NOT NULL AND @pat_offset >= 0 AND

@sizeneeded IS NOT NULL
READTEXT pub_info.pr_info @mytextptr @pat_offset @sizeneeded

COMMIT TRAN

NOTE
If you run the preceding query using SQL Query Analyzer, you might not see the entire text result. SQL Query
Analyzer has a default limit of 256 characters that can be returned for a text column. You can use the Tools/Options
command and go to the Results tab to change this limit.

The offset returned by PATINDEX and the offset used by READTEXT are unfortunately not consistent.
READTEXT treats the first character as offset 0. (This makes sense because you can think of an offset as how many
characters you have to move to get to the desired position—to get to the first character, you don't need to move at
all.) But PATINDEX returns the value in terms of position, not really as an offset, and so the first character for it
would be 1. You could actually call this a minor bug. But people have adapted to it, so changing it would cause more
problems than it would solve at this point. You should assume that this acts as intended and adjust for it. You need to
fix this discrepancy by taking the result of PATINDEX and subtracting 1 from it. PATINDEX returns 0 if the pattern
is not found. Since we subtract 1 from the value returned by PATINDEX, if the pattern was not found, the variable
@pat_offset would be -1. We simply check that @pat_offset is not negative.

You also need to specify how many bytes you want to read. If you want to read from that point to the end of the
column, for example, you can take the total length as returned from datalength and subtract the starting position, as
shown in the preceding example. You cannot simply specify a buffer that you know is large enough to read the rest of
the column into; that will result in error 7124:
This document is created with the unregistered version of CHM2PDF Pilot

The offset and length specified in the READTEXT command is greater

than the actual data length of %d.

If you could always perform a single READTEXT to handle your data, you'd probably not use it; instead, you could
use SELECT. You need to use READTEXT when a text column is too long to reasonably bring back with just one
statement. For example, the text size of the pr_info field for publisher 1622 is 18,518 bytes. You can't select this
value in an application such as SQL Query Analyzer because it's longer than the maximum expected row length for a
result set, so it would be truncated. But you can set up a simple loop to show the text in pieces. To understand this
process, you need to be aware of the system function @@TEXTSIZE, which returns the maximum amount of text or
image data that you can retrieve in a single statement. Of course, you need to make sure that the buffer in your
application will also be large enough to accept the text. You can read chunks smaller than the @@TEXTSIZE limit,
but not larger.
NOTE
The value for @@TEXTSIZE is unrelated to the setting in SQL Query Analyzer for the maximum number of
characters discussed in the previous note. @@TEXTSIZE is a server option, while the character limit controls the
client tool's display.

You can change the value of @@TEXTSIZE for your connection by using SET TEXTSIZE n. The default value for
@@TEXTSIZE is 64 KB. You should read chunks whose size is based on the amount of space available in your
application buffer and based on the network packet size so that the text will fit in one packet, with an allowance for
some additional space for metadata. For example, with the default network packet size of 4096 bytes (4 KB), a good
read size would be about 4000 bytes, assuming that your application could deal with that size. You should also be
sure that @@TEXTSIZE is at least equal to your read size. You can either check to determine its size or explicitly set
it, as we do in the example below. Also notice the handy use of the CASE statement for a variable assignment to
initialize the @readsize variable to the smaller of the total length of the column and the value of @@TEXTSIZE. In
this example, I make the read size only 100 characters so that it displays easily in SQL Query Analyzer or in a similar
query window. But this is too small for most applications and is used here for illustration only.
-- READTEXT in a loop to read chunks of text.
-- Instead of using HOLDLOCK, use SET TRANSACTION ISOLATION LEVEL
-- REPEATABLE READ (equivalent). Then set it back when done but
-- be sure to do so in a separate batch.
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
SET TEXTSIZE 100 —- Just for illustration; too small for
—- real world. 4000 would be a better value.
BEGIN TRAN

DECLARE @mytextptr varbinary(16), @totalsize int,

@lastread int, @readsize int

SELECT
@mytextptr=TEXTPTR(pr_info), @totalsize=DATALENGTH(pr_info),
@lastread=0,
-- Set the readsize to the smaller of the @@TEXTSIZE setting
-- and the total length of the column
@readsize=CASE WHEN (@@TEXTSIZE < DATALENGTH(pr_info)) THEN
@@TEXTSIZE ELSE DATALENGTH(pr_info) END
FROM pub_info WHERE pub_id='1622'

IF @mytextptr IS NOT NULL AND @readsize > 0

WHILE (@lastread < @totalsize)
BEGIN
READTEXT pub_info.pr_info @mytextptr @lastread @readsize
IF (@@error <> 0)
BREAK -- Break out of loop if an error on read
-- Change offset to last char read
SELECT @lastread=@lastread + @readsize
-- If read size would go beyond end, adjust read size
This document is created with the unregistered version of CHM2PDF Pilot

IF ((@readsize + @lastread) > @totalsize)

SELECT @readsize=@totalsize - @lastread
END

COMMIT TRAN
GO
—- Set it back, but in a separate batch
SET TRANSACTION ISOLATION LEVEL READ COMMITTED

Notice that in this example I need to ensure not only that the text pointer is still valid when we get to READTEXT but
also that the column is not changed between iterations of READTEXT. If another connection simply updated the text
in place, the text pointer would still be valid, although the read would be messed up since the contents and length were
changed. I could use HOLDLOCK both on the READTEXT statement as well as for protecting the text pointer. But
for illustration, I instead changed the isolation level to REPEATABLE READ.
NOTE
If the pub_info table in your pubs database doesn't have any text data in the pr_info column, you can load it using a
supplied script in the install directory of your SQL Server installation folder. The script called pubtext.bat requires two
parameters: your sa password and the name of the SQL Server. There is also a script to load in image data for the
logo column of pub_info, called pubimage.bat. Both of these scripts use the undocumented command line utility
called textcopy. (Although textcopy will allow image data to be copied into a SQL Server image column, there are
no supplied facilities for displaying image data graphically. You'll need to use a client tool that provides this capability
in order to "see" the graphics.)

Copying Text to Sequenced varchar Columns

As you've seen, the text datatype is sometimes awkward to work with. Many functions don't operate on text, stored
procedures are limited in what they can do with text, and some tools don't deal with it well. If the amount of text isn't
too big—and you get to be the judge of what is too big for you—you can covert a text column to a set of varchar
columns. In SQL Server 2000, varchar columns can be up to 8000 bytes. Although the code that follows can
produce columns of that size, that might still be too big to work with effectively. All of SQL Server's string functions
will work on these columns, but if you want to print out each column, you might want to find something smaller that fits
your needs better. The code below produces columns of length 200. You can take this code and substitute
varchar(8000) or any other length up to 8000 for varchar(200).

Like the previous example, the following procedure operates on the pub_info table from the pubs database. For a
given pub_id, the procedure takes the text column of the row and does successive READTEXT operations in chunks
of 200 until the entire text column has been read. After creating the procedure, we use a cursor to iterate for each row
of pub_info and copy the text column into multiple rows of a temporary table. When we're done for a given pub_id,
we take the temporary table's contents and add it to the permanent table. We truncate the temporary table and move
on to the next pub_id value.
-- copy_text_to_varchar.sql
-- Proc get_text does READTEXT in a loop to read chunks of text
-- no larger than 200, or the column size or @@TEXTSIZE, and
-- produces as many rows as necessary to store the text column
-- as a series of sequenced varchar(200) rows
CREATE PROC get_text @pub_id char(4)
AS

DECLARE @mytextptr varbinary(16), @totalsize int, @lastread int,

@readsize int
-- Use a TRAN and HOLDLOCK to ensure that textptr and text are
-- constant during the iterative reads
BEGIN TRAN
SELECT @mytextptr=TEXTPTR(pr_info), @totalsize=DATALENGTH(pr_info),
@lastread=0,
-- Set the readsize to the smaller of the @@TEXTSIZE settings,
This document is created with the unregistered version of CHM2PDF Pilot

-- 200, and the total length of the column

@readsize=CASE WHEN (200 < DATALENGTH(pr_info)) THEN 200
ELSE DATALENGTH(pr_info) END
FROM pub_info (HOLDLOCK) WHERE pub_id=@pub_id

-- If debugging, uncomment this to check values

-- SELECT @mytextptr, @totalsize, @lastread, @readsize

-- Do READTEXT in a loop to get next 200 characters until done

IF @mytextptr IS NOT NULL AND @readsize > 0
WHILE (@lastread < @totalsize)
BEGIN
-- If readsize would go beyond end, adjust readsize
IF ((@readsize + @lastread) > @totalsize)
SELECT @readsize = @totalsize - @lastread

-- If debugging, uncomment this to check values

-- SELECT 'valid ptr?'=textvalid('pub_info.pr_info',
-- @mytextptr), 'totalsize'=@totalsize,
-- 'lastread'=@lastread, 'readsize'=@readsize

READTEXT pub_info.pr_info @mytextptr @lastread @readsize

IF (@@error <> 0)
BREAK -- Break out of loop if an error on read
-- Change offset to last char read
SELECT @lastread=@lastread + @readsize

END

COMMIT TRAN
GO

IF EXISTS (SELECT * FROM tempdb.dbo.sysobjects

WHERE name='##mytmptext' AND type='U')
DROP TABLE ##mytmptext
GO

-- Intermediate temp table that READTEXT will use.

-- This table is truncated after each pub_id value, so the Identity
-- property sequences the rows for each publisher separately.
CREATE TABLE ##mytmptext
(
seq_no int IDENTITY,
text_chunk text
)
GO

IF EXISTS (SELECT * FROM sysobjects

WHERE name='newprinfo' AND type='U')
DROP TABLE newprinfo
GO

-- This is the new table that pub_info is copied to.

-- It keeps chunks of text and is sequenced for each pub_id.
CREATE TABLE newprinfo
(
pub_id char(4) NOT NULL,
seq_no int NOT NULL,
text_chunk varchar(200),
CONSTRAINT PK PRIMARY KEY (pub_id, seq_no)
)
GO

-- Having created the procedure get_text, iterate for each pub_id

-- value, temporarily sequencing them in the temp table. Once done
This document is created with the unregistered version of CHM2PDF Pilot

-- for a given pub_id, copy them to the new permanent table. Then
-- truncate the temp table for use with reseeded Identity for next
-- pub_id row.
DECLARE @pub_id char(4)
DECLARE iterate_prinfo CURSOR FOR
SELECT pub_id FROM pub_info ORDER BY pub_id

OPEN iterate_prinfo
FETCH NEXT FROM iterate_prinfo INTO @pub_id
BEGIN
TRUNCATE TABLE ##mytmptext

INSERT ##mytmptext
EXEC get_text @pub_id

INSERT newprinfo (pub_id, seq_no, text_chunk)

SELECT @pub_id, SEQ_NO,
CONVERT(varchar(200), TEXT_CHUNK)
FROM ##mytmptext ORDER BY SEQ_NO

FETCH NEXT FROM iterate_prinfo INTO @pub_id

END

CLOSE iterate_prinfo
DEALLOCATE iterate_prinfo
GO

-- Simply verify contents of the new table

SELECT * FROM newprinfo ORDER BY pub_id,seq_no
GO

A subset of the output looks like the following:

pub_id seq_no text_chunk
------ ------ -----------------------------------------------
0736 1 This is sample text data for New Moon Books,
publisher 0736 in the pubs database. New Moon
Books is located in Boston, Massachusetts.
This is sample text data for New Moon Books,
publisher 0736 in
0736 2 the pubs database. New Moon Books is located in
Boston, Massachusetts.
This is sample text data for New Moon Books,
publisher 0736 in the pubs database. New Moon
Books is located in Boston, Massach
0736 3 usetts.
This is sample text data for New Moon Books,
publisher 0736 in the pubs database. New Moon
Books is located in Boston, Massachusetts.
This is sample text data for New Moon Books,
publish

UPDATETEXT
UPDATETEXT is a big improvement for text processing. If you had only WRITETEXT, you'd need to completely
rewrite the entire column to make even a minor change. UPDATETEXT lets you work with text in pieces to insert,
overwrite, or append data. Or you can copy data from another text column and append it or overwrite the column
with it. Because of its additional capability and flexibility, the syntax for UPDATETEXT is a bit more complex:

UPDATETEXT table_name.dest_column_name dest_text_ptr

This document is created with the unregistered version of CHM2PDF Pilot

offset delete_length [inserted_data |

table_name.src_column_name src_text_ptr ]

The destination column name and text pointer parameters point to the column that you will be updating; these
parameters are required. As you would with WRITETEXT, you should use the UPDLOCK hint to protect the text
pointer from becoming invalid and to serialize access to the text pages to prevent a conversion deadlock. The source
column name parameters are used only when you copy data from another text column. Otherwise, you directly
include in that spot the data you'll be adding or you omit the parameter if you're deleting data. The offset is the
position at which you start your data modification. It should be NULL if you're appending to the current contents and
0 if you're starting from the beginning of the column. The delete_length parameter tells you how many characters (if
any) to delete starting from the offset parameter. Use NULL for this parameter if you want to delete all contents from
the offset up to the end of the column, and use 0 if you want to delete no bytes. As with READTEXT, the first
character of the column is considered to have a 0 offset.

UPDATETEXT can do everything WRITETEXT can do, and much more. So you might choose to use only
READTEXT and UPDATETEXT and forget about WRITETEXT. (WRITETEXT existed in versions of SQL Server
before UPDATETEXT appeared; it is maintained for backward compatibility, but there isn't much need for it now.)

Following are some examples that illustrate the use of UPDATETEXT better than further explanation.

EXAMPLE 1

Use UPDATETEXT to completely replace the contents of a column:

-- Use UPDATETEXT to completely overwrite a text column.
-- Alternative to WRITETEXT.
DECLARE @mytextptr varbinary(16)
BEGIN TRAN

SELECT @mytextptr=TEXTPTR(pr_info) FROM pub_info (UPDLOCK) WHERE

pub_id='9999'
IF @mytextptr IS NOT NULL
UPDATETEXT pub_info.pr_info @mytextptr 0 NULL
'New text for 9999'

COMMIT TRAN

EXAMPLE 2

Use UPDATETEXT to delete characters off the end; first notice that publisher 0877, Binnet & Hardley, has the
following contents in the text column pr_info:
This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.
This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.
This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.
This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.
This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.
This document is created with the unregistered version of CHM2PDF Pilot

Because the text is repeated several times, we'll delete all characters that follow the first occurrence of D.C.:
DECLARE @mytextptr varbinary(16), @pat_offset int
BEGIN TRAN
SELECT @mytextptr=TEXTPTR(pr_info),
@pat_offset=PATINDEX('%D.C.%', pr_info)-1+4
-- For offset, subtract 1 for offset adjust but add 4 for
-- length of "D.C."
FROM pub_info (UPDLOCK) WHERE pub_id='0877'

IF @mytextptr IS NOT NULL AND @pat_offset >= 4

UPDATETEXT pub_info.pr_info @mytextptr @pat_offset NULL

COMMIT TRAN

The column now has these contents (only):

This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.

EXAMPLE 3

With the small amount of text here, it wouldn't be bad to simply rewrite the column with new text. But if this were a
large text column (you could literally store the contents of War and Peace in a single text column), it would be
extremely inefficient to rewrite the entire column just to make a minor change. In this example, we'll add the text
Kimberly Tripp is president of the company. to the current contents. We'll use UPDATETEXT to append text to
the column:
DECLARE @mytextptr varbinary(16)
BEGIN TRAN
SELECT @mytextptr=TEXTPTR(pr_info) FROM pub_info (UPDLOCK)
WHERE pub_id='0877'

IF @mytextptr IS NOT NULL

UPDATETEXT pub_info.pr_info @mytextptr NULL NULL
'Kimberly Tripp is president of the company.'

COMMIT TRAN

And the result:

This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.Kimberly Tripp is president of the company.

That worked exactly as specified, but you might wish we had skipped a line and then included a tab before adding
the new sentence. We can easily add both a vertical and a horizontal tab, as you can see in the next example.

EXAMPLE 4

Use UPDATETEXT to insert some characters:

DECLARE @mytextptr varbinary(16), @pat_offset int,
@mystring char(2)
BEGIN TRAN
This document is created with the unregistered version of CHM2PDF Pilot

SELECT
@mystring=char(13) + CHAR(9), -- Vertical tab is code point 13.
-- Tab is 9.
@pat_offset=PATINDEX('%Kim%', pr_info)-1,
@mytextptr=TEXTPTR(pr_info) FROM pub_info (UPDLOCK)
WHERE pub_id='0877'

IF @mytextptr IS NOT NULL AND @pat_offset >= 0

UPDATETEXT pub_info.pr_info @mytextptr @pat_offset 0
@mystring

COMMIT TRAN

And the result:

This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.
Kimberly Tripp is president of the company.

EXAMPLE 5

Oops! We just learned that the president has gotten married and changed her last name from Tripp to
Tripp-Simonnet. We need to fix that. Use UPDATETEXT for search and replace:
-- UPDATETEXT for Search and Replace
DECLARE @mytextptr varbinary(16), @pat_offset int,
@oldstring varchar(255), @newstring varchar(255),
@sizeold int

BEGIN TRAN
SELECT @oldstring='Tripp', @newstring='Tripp-Simonnet'

SELECT @sizeold=DATALENGTH(@oldstring),
@pat_offset=PATINDEX('%' + @oldstring + '%', pr_info)-1,
@mytextptr=TEXTPTR(pr_info)
FROM pub_info (UPDLOCK) WHERE pub_id='0877'

IF @mytextptr IS NOT NULL AND @pat_offset >= 0

UPDATETEXT pub_info.pr_info @mytextptr @pat_offset @sizeold
@newstring

COMMIT TRAN

And the result:

This is sample text data for Binnet & Hardley, publisher 0877 in
the pubs database. Binnet & Hardley is located in Washington,
D.C.
Kimberly Tripp-Simonnet is president of the company.

We used variables above and figured lengths and offsets using SQL Server's built-in functions. By doing this, we
ensured that the procedure is pretty generic and that it can deal with changing the string to another string that is either
longer or shorter than the original.

EXAMPLE 6

Suppose we want to append the contents of the text for publisher Scootney (pub_id 9952) to the text for Binnet (
This document is created with the unregistered version of CHM2PDF Pilot

pub_id 0877). If we didn't have this option in UPDATETEXT, it would be necessary to bring all that text back to the
client application, append it, and then send it back to the server. Over a slow network such as the Internet, this would
not be practical if the text columns were large. But with UPDATETEXT, the whole operation is done on the server.
In this example, notice that we protect the text pointer for the target with UPDLOCK, since we'll be updating that
row. We can use HOLDLOCK for the source row since we're reading it only and we want to ensure that it hasn't
changed.

We'll use UPDATETEXT to copy and append one text column to another:
-- UPDATETEXT to copy and append another text column
DECLARE @target_textptr varbinary(16),
@source_textptr varbinary(16)
BEGIN TRAN

SELECT @target_textptr=TEXTPTR(pr_info) FROM pub_info (UPDLOCK)

WHERE pub_id='0877'
SELECT @source_textptr=TEXTPTR(pr_info) FROM pub_info (HOLDLOCK)
WHERE pub_id='9952'

IF @target_textptr IS NOT NULL AND @source_textptr IS NOT NULL

UPDATETEXT pub_info.pr_info @target_textptr NULL NULL
pub_info.pr_info @source_textptr

COMMIT TRAN

Text-in-Row Data
As discussed in Chapter 6, you can enable a table to store LOB data in the row itself. Most of the examples I've
shown, as well as the details regarding the READTEXT, WRITETEXT, and UPDATEXT, apply equally to LOB
data stored in a row or out of a row. However, the previous discussion of locking the text pointer applies only to
LOB data that is stored outside of the data row. For tables that have text in row enabled, the locking issues with the
text pointer are slightly different. When a text pointer is acquired for text data in a table with text in row enabled, the
data row is locked with a share lock if the isolation level of the transaction is higher than READ UNCOMMITTED,
and the database is not in read-only or single-user mode. The lock ensures that nobody else can modify or delete the
row while you have a text pointer on a text value from that row. This lock is released when the text pointer becomes
invalid. Since at the end of the transaction all text pointers are invalid, the locks are all released. You'll see a few more
details about what kind of operations can invalidate a text pointer for in-row LOB data later in this section.

To illustrate the automatic locking of the row with in-row text data, let's see some basic examples. The share locks
on rows with in-row text are held until the end of a transaction, and this means that we cannot get a text pointer for
in-row text unless we are in a user defined transaction. This example shows an attempt to acquire a text pointer
without an explicit transaction:
CREATE TABLE t1 (c1 int, c2 text)
EXEC sp_tableoption 't1', 'text in row', 'ON'
INSERT INTO t1 SELECT 1, 'hello there'
GO

DECLARE @ptrval varbinary(16)

SELECT @ptrval = TEXTPTR(c2) FROM t1 WHERE c1 = 1
READTEXT t1.c2 @ptrval 0 1

RESULTS:
Server: Msg 7101, Level 16, State 1, Line 3
You cannot use a text pointer for a table with option 'text in row'
set to ON.
Server: Msg 7133, Level 16, State 1, Line 4
NULL textptr (text, ntext, or image pointer) passed to
READTEXT function.
This document is created with the unregistered version of CHM2PDF Pilot

However, attempting to do the same text pointer assignment and READTEXT inside a user-defined transaction
succeeds:
BEGIN TRAN
DECLARE @ptrval varbinary(16)
SELECT @ptrval = TEXTPTR(c2) FROM t1 WHERE c1 = 1
READTEXT t1.c2 @ptrval 0 1
COMMIT TRAN

To see that the shared locks are taken and held automatically, we can execute the same code shown earlier for the
text data in the pub_info table, which was not stored in the row itself. If you actually ran that earlier example, the row
with pub_id 9999 is now gone, so I'll change the code to look for a row with pub_id 1756. I also must first enable
the text in row option for the pub_info table and then use an explicit transaction.
-- WRITETEXT with text-in-row data
EXEC sp_tableoption pub_info, 'text in row', 'ON'
GO

BEGIN TRAN
DECLARE @mytextptr varbinary(16)
SELECT @mytextptr=TEXTPTR(pr_info)
FROM pub_info WHERE pub_id='1756'
WAITFOR DELAY '00:00:20'
-- During the delay, you can attempt to delete
-- this row in another Query Analyzer connection:
-- DELETE FROM pub_info WHERE pub_id='1756'
IF @mytextptr IS NOT NULL
WRITETEXT pub_info.pr_info @mytextptr 'Hello Again'
COMMIT TRAN

If the isolation level of the transaction is READ UNCOMMITTED, any text pointer obtained from a table with text
in row enabled is considered read-only and cannot be used to update LOB values. They can be used only to read the
data. The example below creates a table with in-row text data and then sets the transaction isolation level to READ
UNCOMMITTED. You can see that the READTEXT succeeds but the WRITETEXT does not.
CREATE TABLE t1 (c1 int, c2 text)
EXEC sp_tableoption 't1', 'text in row', 'ON'
INSERT INTO t1 SELECT 1, 'hello there'
GO

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED

GO

BEGIN TRAN
DECLARE @ptr varbinary(16)
SELECT @ptr=textptr(c2) FROM t1 WHERE c1=1
READTEXT t1.c2 @ptr 0 5
COMMIT TRAN
GO

BEGIN TRAN
DECLARE @ptr varbinary(16)
SELECT @ptr=textptr(c2) FROM t1 WHERE c1=1
WRITETEXT t1.c2 @ptr 'xx'
COMMIT TRAN
GO
This document is created with the unregistered version of CHM2PDF Pilot

This is the error message returned by the WRITETEXT operation:

Server: Msg 7106, Level 16, State 1, Line 5
You cannot update a blob with a read-only text pointer
The statement has been terminated.

Note that the message uses the term BLOB, which should refer just to image fields; in my discussion, I've been using
the term LOB, which refers to text and ntext as well as image.

Invalidating Text Pointers As you saw earlier, if a table was not enabled for in-row text storage, no row lock is
held for the text pointer. The only way you saw to lock the data row was to raise the isolation level to REPEATABLE
READ. The drawback of using the isolation level to prolong the lock is that there is no way to choose to release the
lock earlier. When working with LOB data in a table that has text in row enabled, you do have control over when
the lock is released because it will be released whenever the text pointer becomes invalidated. I mentioned above that
the end of a transaction will invalidate a text pointer, but that is not the only way.

Text pointers for in-row text data are also invalidated when certain DDL operations are performed on the table. Here
are the DDL statements that invalidate your text pointers:
 CREATE CLUSTERED INDEX
 DROP CLUSTERED INDEX
 ALTER TABLE
 DROP TABLE
 TRUNCATE TABLE
 sp_tableoption with the text in row option
 sp_indexoption

Finally, you can invalidate a text pointer for in-row text data using a special stored procedure, sp_invalidate_textptr
, which takes an argument of a text pointer.

You might need to force invalidation of text pointers because SQL Server sets an upper limit on the number of
pointers for in-row text data. A limit is needed because SQL Server maintains internal memory for each valid text
pointer, and we want to limit the amount of memory that can be used in this way. You can have up to 1024 valid text
pointers per transaction per database. If your transaction spans two databases, you can have 1024 text pointers in
each database. Within a transaction, obtaining a text pointer twice on the same LOB data gives you the same text
pointer structure and will not count against your limit of 1024. Note that there is no limit on the number of pointers you
can have for LOB data that is not stored in the row.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
In this chapter, we looked at two aspects of Transact-SQL programming that go beyond the set-based DML
operations of SELECT, INSERT, UPDATE, and DELETE. First, I described server cursors in detail. Cursors are
important tools that help you bridge the impedance mismatch between the set-oriented world of SQL Server and
traditional record-oriented programming languages. When used properly, cursors are invaluable. When overused or
used inappropriately, they can cause you to have a bad experience with SQL Server. SQL Server provides fully
scrollable, updateable cursors.

SQL Server has four main cursor models: static, keyset, dynamic, and forward-only. Cursors can be accessed via
syntax used directly in Transact-SQL or via function calls used in the programming interfaces to SQL Server (ODBC,
OLE DB, and so on). These two types of cursors are called Transact-SQL cursors and API server cursors. This
chapter focused on the Transact-SQL syntax, but the key points are relevant no matter what cursor interface you use
with SQL Server.

Second, we looked at the processing of data that might be too large to store in the actual table rows. SQL Server
provides the three datatypes—text, ntext, and image—to support LOB (large object) data, which can store up to 2
GB worth of data. You can use three special operators with LOB data: READTEXT, WRITETEXT, and
UPDATETEXT. Because no datatypes can store this very large object, most manipulation of LOB data is done using
text pointers. The three LOB operators take a text pointer as an argument and can allow you to access and
manipulate only a subset of the data.
This document is created with the unregistered version of CHM2PDF Pilot

34

Part IV

Performance and Tuning

This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 14

Locking
Locking is a crucial function of any multiple-user database system, including Microsoft SQL Server. As you know,
SQL Server manages multiple users simultaneously and ensures that all transactions observe the properties of the
specified isolation level. At the highest isolation level, Serializable, SQL Server must make the multiple-user system
yield results that are indistinguishable from those of a single-user system. It does this by automatically locking data to
prevent changes made by one user from unexpectedly affecting work being done by another user on the same
database at the same time.
This document is created with the unregistered version of CHM2PDF Pilot

34

The Lock Manager

SQL Server can lock data using several different modes. For example, read operations acquire shared locks and
write operations acquire exclusive locks. Update locks are acquired during the initial portion of an update operation
when the data is read. The SQL Server lock manager acquires and releases these locks. It also manages compatibility
between lock modes, resolves deadlocks, and escalates locks if necessary. It controls locks on tables, on the pages
of a table, on index keys, and on individual rows of data. Locks can also be held on system data—data that's private
to the database system, such as page headers and indexes.

The lock manager provides two separate locking systems. The first system affects all fully shared data and provides
row locks, page locks, and table locks for tables, data pages, text pages, and leaf-level index pages. The second
system is used internally for index concurrency control, controlling access to internal data structures, and retrieving
individual rows of data pages. This second system uses latches, which are less resource intensive than locks and
provide performance optimization. You could use full-blown locks for all locking, but because of their complexity,
they would slow down the system if they were used for all internal needs. If you examine locks using the sp_lock
system stored procedure or a similar mechanism that gets information from the syslockinfo table, you cannot see
latches—you see only information about locks for fully shared data.

Another way to look at the difference between locks and latches is that locks ensure the logical consistency of the
data and latches ensure the physical consistency. Latching happens when you place a row physically on a page or
move data in other ways, such as compressing the space on a page. SQL Server must guarantee that this data
movement can happen without interference.

The Lock Manager and Isolation Levels

SQL Server supports all four transaction isolation levels specified by ANSI and ISO: Serializable, Repeatable Read,
Read Committed, and Read Uncommitted. (See Chapter 3 and Chapter 12 for details.) To achieve the Serializable
isolation level, you must prevent phantoms because the transaction's behavior must be identical to what would have
occurred had the transaction run on a single-user system. SQL Server provides serializability, which you can set using
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE. To support serializability, SQL Server locks index
ranges using a special type of lock called a key-range lock. Such locks are held until the end of the transaction in
order to prevent phantoms. If no index exists, the lock manager uses a table lock to guarantee serializability.

The lock manager provides fairly standard two-phase locking services. Although the names are similar, two-phase
locking (2PL) and the two-phase commit (2PC) protocol are not directly related other than by the obvious fact that
2PC must use 2PL services. In 2PL, a transaction has a "growing" phase, during which it acquires locks, and a
"shrinking" phase, during which it releases locks. To achieve serializability, all acquired locks are held until the end of
the transaction and then are dropped all at once.

For a lower isolation level, such as Committed Read, locks can be released sooner—when the use of the object is
completed. For example, if a range of data is being queried in the table, there will probably be shared locks
outstanding. With Committed Read isolation, a shared lock is released as soon as the scan moves off one piece of
data and onto the next. Exclusive locks, on the other hand, are always held until the end of the transaction so that the
transaction can be rolled back if necessary.

With Serializable or Repeatable Read isolation, shared locks must be held until the end of the transaction to
guarantee that the data that was read will not change or that new rows meeting the criteria of the query cannot be
added while the transaction is in progress. Like shared locks, latches are not tied to the boundaries of a transaction
because they are used to provide mutual exclusion (mutex) functionality rather than to directly lock data. For example,
This document is created with the unregistered version of CHM2PDF Pilot

during a row insert in a table with a clustered index, the nearby index page is latched to prevent other inserts from
colliding. The latches are needed to provide mutual exclusion only during long periods of time (that is, periods with
more than a few instruction cycles).

Spinlocks
For shorter-term needs, SQL Server achieves mutual exclusion using a latch implemented with a spinlock. Spinlocks
are used purely for mutual exclusion and never to lock user data. They are even more lightweight than latches, which
are lighter than the full locks used for data and index leaf pages. The spinlock is the only functionality in SQL Server in
which processor-specific assembly language is used. A spinlock is implemented in a few lines of assembly language
specific to each processor type (such as x86/Pentium or Alpha). The requester of a spinlock repeats its request if the
lock is not immediately available. (That is, the requester "spins" on the lock until it is free.)

Spinlocks are often used as mutexes within SQL Server when a resource is usually not busy. If a resource is busy,
the duration of a spinlock is short enough that retrying is better than waiting and then being rescheduled by the
operating system, which results in context switching between threads. The savings in context switches more than
offsets the cost of spinning as long as you don't have to spin too long. Spinlocks are used for situations in which the
wait for a resource is expected to be brief (or if no wait is expected).

Deadlocks
A deadlock occurs when two processes are waiting for a resource and neither process can advance because the
other process prevents it from getting the resource. A true deadlock is a catch-22 in which, without intervention,
neither process can ever progress. When a deadlock occurs, SQL Server intervenes automatically.
NOTE
A simple wait for a lock is not a deadlock. When the process that's holding the lock completes, the waiting process
gets the lock. Lock waits are normal, expected, and necessary in multiple-user systems.

In SQL Server, two main types of deadlocks can occur: a cycle deadlock and a conversion deadlock. Figure 14-1
shows an example of a cycle deadlock. Process A starts a transaction, acquires an exclusive table lock on the
authors table, and requests an exclusive table lock on the publishers table. Simultaneously, process B starts a
transaction, acquires an exclusive lock on the publishers table, and requests an exclusive lock on the authors table.
The two processes become deadlocked—caught in a "deadly embrace." Each process holds a resource needed by
the other process. Neither can progress, and, without intervention, both would be stuck in deadlock forever. You can
actually generate the deadlock using the SQL Query Analyzer, as follows:
1. Open a query window, and change your database context to the pubs database. Execute the following batch
for process A:
BEGIN TRAN
UPDATE authors SET contract = 0
GO

2. Open a second window, and execute this batch for process B:

BEGIN TRAN
UPDATE publishers SET city = 'Redmond', state = 'WA'
GO

3. Go back to the first window, and execute this update statement:

UPDATE publishers SET city = 'New Orleans', state = 'LA'
GO
This document is created with the unregistered version of CHM2PDF Pilot

At this point, the query should block. It is not deadlocked yet, however. It is waiting for a lock on the
publishers table, and there is no reason to suspect that it won't eventually get that lock.
4. Go back to the second window, and execute this update statement:
UPDATE authors SET contract = 1
GO

At this point, a deadlock occurs. The first connection will never get its requested lock on the publishers table
because the second connection will not give it up until it gets a lock on the authors table. Since the first
connection already has the lock on the authors table, we have a deadlock. One of the processes received the
following error message. (Of course, the actual process ID reported will probably be different.)
Server: Msg 1205, Level 13, State 1, Line 1
Transaction (Process ID 55) was deadlocked on {lock} resources with
another process and has been chosen as the deadlock victim. Rerun the
transaction.

Figure 14-2 shows an example of a conversion deadlock. Process A and process B each hold a shared lock on the
same page within a transaction. Each process wants to promote its shared lock to an exclusive lock but cannot do so
because of the other process's lock. Again, intervention is required.

Figure 14-1. A cycle deadlock resulting from each of two processes holding a resource needed by the other.

Figure 14-2. A conversion deadlock resulting from two processes wanting to promote their locks on the same
resource within a transaction.
This document is created with the unregistered version of CHM2PDF Pilot

SQL Server automatically detects deadlocks and intervenes through the lock manager, which provides deadlock
detection for regular locks. In SQL Server 2000, deadlocks can also involve resources other than locks. For
example, if process A is holding a lock on Table1 and is waiting for memory to become available and process B has
some memory it can't release until it acquires a lock on Table1, the processes will deadlock. Threads and
communication buffers can also be involved in deadlocks. Latches are not involved in deadlock detection because
SQL Server uses deadlock-proof algorithms when it acquires latches. When SQL Server detects a deadlock, it
terminates one process's batch, rolling back the active transaction and releasing all that process's locks to resolve the
deadlock.

In SQL Server 2000, a separate thread called LOCK_MONITOR checks the system for deadlocks every 5
seconds. The lock monitor also uses an internal counter called a deadlock detection counter to determine whether to
check the system for deadlocks more often. The deadlock detection counter starts at a value of 3 and is reset to 3 if a
deadlock occurs. If the LOCK_MONITOR thread finds no deadlocks when it checks at the end of its 5-second
cycle, it decrements the deadlock detection counter. If the counter has a value greater than 0, the lock manager
requests that the lock monitor also check all the locks for a deadlock cycle if a process requests a lock resource and
is blocked. Thus, after 20 seconds of not finding any deadlocks, the deadlock detection counter is 0 and the lock
manager stops requesting deadlock detection every time a process blocks on a lock. The deadlock detection counter
stays at 0 most of the time and the checking for deadlocks happens only at the 5-second intervals of the lock monitor.

This LOCK_MONITOR thread checks for deadlocks by inspecting the list of waiting locks for any cycles, which
indicate a circular relationship between processes holding locks and processes waiting for locks. SQL Server
attempts to choose as the victim the process that would be the least expensive to roll back, considering the amount of
work the process has already done. However, certain operations are marked as golden, or unkillable, and cannot be
chosen as the deadlock victim. For example, a process involved in rolling back a transaction cannot be chosen as a
deadlock victim because the changes being rolled back could be left in an indeterminate state, causing data
corruption.

Using the SET DEADLOCK_PRIORITY LOW | NORMAL statement, you can make a process sacrifice itself as
the victim if a deadlock is detected. If a process has a deadlock priority of LOW, it terminates when a deadlock is
detected even if it is not the process that closed the loop. However, there is no counterpart SET option to set a
deadlock priority to HIGH. As much as you might want your own processes to always come out the winner in a
deadlock situation, this feature has not yet been implemented in SQL Server.
NOTE
The lightweight latches and spinlocks used internally do not have deadlock detection services. Instead, deadlocks on
latches and spinlocks are avoided rather than resolved. Avoidance is achieved via strict programming guidelines used
by the SQL Server development team. These lightweight locks must be acquired in a hierarchy, and a process must
not have to wait for a regular lock while holding a latch or spinlock. For example, one coding rule is that a process
holding a spinlock must never directly wait for a lock or call another service that might have to wait for a lock, and a
request can never be made for a spinlock that is higher in the acquisition hierarchy. By establishing similar guidelines
for your development team for the order in which SQL Server objects are accessed, you can go a long way toward
avoiding deadlocks in the first place.

In the example in Figure 14-1, the cycle deadlock could have been avoided if the processes had decided on a
protocol beforehand—for example, if they had decided to always access the authors table first and the publishers
table second. Then one of the processes would get the initial exclusive lock on the table being accessed first, and the
other process would wait for the lock to be released. One process waiting for a lock is normal and natural.
Remember, waiting is not a deadlock.

You should always try to have a standard protocol for the order in which processes access tables. If you know that
the processes might need to update the row after reading it, they should initially request an update lock, not a shared
lock. If both processes request an update lock rather than a shared lock, the process that is granted an update lock is
assured that the lock can later be promoted to an exclusive lock. The other process requesting an update lock has to
wait. The use of an update lock serializes the requests for an exclusive lock. Other processes needing only to read the
data can still get their shared locks and read. Since the holder of the update lock is guaranteed an exclusive lock, the
This document is created with the unregistered version of CHM2PDF Pilot

deadlock is avoided. We'll look in more detail at the compatibility of locks later in this chapter; additional information
on deadlocks is presented in Chapter 16.

By the way, the time that your process holds locks should be minimal so other processes don't wait too long for
locks to be released. Although you don't usually invoke locking directly, you can influence locking by keeping
transactions as short as possible. For example, don't ask for user input in the middle of a transaction. Instead, get the
input first and then quickly perform the transaction.
This document is created with the unregistered version of CHM2PDF Pilot

34

Lock Types for User Data

Now we'll examine four aspects of locking user data. First, we'll look at the mode of locking (the type of lock). I
already mentioned shared, exclusive, and update locks, and I'll go into more detail about these modes as well as
others. Next, we'll look at the granularity of the lock, which specifies how much data is covered by a single lock. This
can be a row, a page, an index key, a range of index keys, an extent, or an entire table. The third aspect of locking is
the duration of the lock. As mentioned earlier, some locks are released as soon as the data has been accessed and
some locks are held until the transaction commits or rolls back. For example, cursor scroll locks are held until a new
FETCH operation is executed. The fourth aspect of locking concerns the ownership of the lock (the scope of the
lock). Locks can be owned by a session, a transaction, or a cursor.

Lock Modes
SQL Server uses several locking modes, including shared locks, exclusive locks, update locks, and intent locks.

Shared Locks
Shared locks are acquired automatically by SQL Server when data is read. Shared locks can be held on a table, a
page, an index key, or an individual row. Many processes can hold shared locks on the same data, but no process
can acquire an exclusive lock on data that has a shared lock on it (unless the process requesting the exclusive lock is
the same process as the one holding the shared lock). Normally, shared locks are released as soon as the data has
been read, but you can change this by using query hints or a different transaction isolation level.

Exclusive Locks
SQL Server automatically acquires exclusive locks on data when it is modified by an insert, update, or delete
operation. Only one process at a time can hold an exclusive lock on a particular data resource; in fact, as you'll see
when we discuss lock compatibility, no locks of any kind can be acquired by a process if another process has the
requested data resource exclusively locked. Exclusive locks are held until the end of the transaction. This means that
the changed data is normally not available to any other process until the current transaction commits or rolls back.
Other processes can decide to read exclusively locked data by using query hints.

Update Locks
Update locks are really not a separate kind of lock; they are a hybrid between shared and exclusive locks. They are
acquired when SQL Server executes a data modification operation but first needs to search the table to find the
resource that will be modified. Using query hints, a process can specifically request update locks, and in that case the
update locks prevent the conversion deadlock situation presented in Figure 14-2.

Update locks provide compatibility with other current readers of data, allowing the process to later modify data with
the assurance that the data hasn't been changed since it was last read. An update lock is not sufficient to allow you to
change the data—all modifications require that the data resource being modified have an exclusive lock. An update
lock acts as a serialization gate to queue future requests for the exclusive lock. (Many processes can hold shared
locks for a resource, but only one process can hold an update lock.) As long as a process holds an update lock on a
resource, no other process can acquire an update lock or an exclusive lock for that resource; instead, another process
requesting an update or exclusive lock for the same resource must wait. The process holding the update lock can
acquire an exclusive lock on that resource because the update lock prevents lock incompatibility with any other
processes. You can think of update locks as "intent-to-update" locks, which is essentially the role they perform. Used
This document is created with the unregistered version of CHM2PDF Pilot

alone, update locks are insufficient for updating data—an exclusive lock is still required for actual data modification.
Serializing access for the exclusive lock lets you avoid conversion deadlocks.

Don't let the name fool you: update locks are not just for update operations. SQL Server uses update locks for any
data modification operation that requires a search for the data prior to the actual modification. Such operations include
qualified updates and deletes, as well as inserts into a table with a clustered index. In the latter case, SQL Server must
first search the data (using the clustered index) to find the correct position at which to insert the new row. While SQL
Server is only searching, it uses update locks to protect the data; only after it has found the correct location and
begins inserting does it escalate the update lock to an exclusive lock.

Intent Locks
Intent locks are not really a separate mode of locking; they are a qualifier to the modes previously discussed. In other
words, you can have intent shared locks, intent exclusive locks, and even intent update locks. Because SQL Server
can acquire locks at different levels of granularity, a mechanism is needed to indicate that a component of a resource
is already locked. For example, if one process tries to lock a table, SQL Server needs a way to determine whether a
row (or a page) of that table is already locked. Intent locks serve this purpose. We'll discuss them in more detail when
we look at lock granularity.

Special Lock Modes

SQL Server offers three additional lock modes: schema stability locks, schema modification locks, and bulk update
locks. When queries are compiled, schema stability locks prevent other processes from acquiring schema modification
locks, which are taken when a table's structure is being modified. A bulk update lock is acquired when the BULK
INSERT command is executed or when the bcp utility is run to load data into a table. In addition, the copy operation
must request this special lock by using the TABLOCK hint. Alternatively, the table can set the table option called
table lock on bulk load to true, and then any bulk copy IN or BULK INSERT operation will automatically request a
bulk update lock. If multiple connections have requested and received a bulk update lock, they can perform parallel
loads into the same table.

Another lock mode that you might notice is the SIX lock. This mode is never requested directly by the lock manager
but is the result of a conversion. If a transaction is holding a shared (S) lock on a resource and later an IX lock is
needed, the lock mode will be indicated as SIX. For example, suppose you are operating at the Repeatable Read
transaction isolation level and you issue the following batch:
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRAN
SELECT * FROM bigtable
UPDATE bigtable
SET col = 0
WHERE keycolumn = 100

Assuming that the table is large, the SELECT statement will acquire a shared table lock. (If there are only a few rows
in bigtable, SQL Server will acquire individual row or key locks.) The UPDATE statement will then acquire a single
exclusive key lock to do the update of a single row, and the X lock at the key level will mean an IX lock at the page
and table level. The table will then show SIX when viewed through sp_lock.

Table 14-1 shows most of the lock modes, as well as the abbreviations used in the output of sp_lock.

Table 14-1. SQL Server lock modes.

Internal Code for Lock
Abrreviation Lock Mode Description
Mode
This document is created with the unregistered version of CHM2PDF Pilot

Allows other processes to

S Shared 4 read but not change the
locked resource.
Prevents another process
from modifying or reading
data in the locked resource
X Exclusive 6
(unless the process is set to
the Read Uncommitted
isolation level).
Prevents other processes
from acquiring an update or
U Update 5 exclusive lock; used when
searching for the data to
modify.
Indicates that a component
of this resource is locked
IS Intent update 7 with a shared lock. This lock
can be acquired only at the
table or page level.
Indicates that a component
of this resource is locked
IU Intent update 8 with an update lock. This
lock can be acquired only at
the table or page level.
Indicates that a component
of this resource is locked
IX Intent Exclusive 9 with an exclusive lock. This
lock can be acquired only at
the table or page level.
Indicates that a resource
intent exclusive holding a
shared lock also has a
SIX Shared with 11
component (a page or
row)locked with an
exclusive lock.
Indicates that a query using
Sch-S Schema stability 2
this table is being compiled.
Indicates that the structure
Sch-M Schema modification 3 fication of the table is being
changed.
Used when a bulk copy
operation is copying data
into a table and the
BU Bulk update 13
TABLOCK hint is being
applied (either manually or
automatically).

Additional locks modes—called key-range locks—are taken only in serializable isolation level for locking ranges of
data. There are nine types of key-range locks, and each has a two-part name: The first part indicates the type of lock
on the range of data between adjacent index keys, and the second part indicates the type of lock on the key itself.
These nine types of key-range locks are described in Table 14-2.
This document is created with the unregistered version of CHM2PDF Pilot

Table 14-2. Key-range locks.

Abbreviation Internal Code for Lock Mode Description
Shared lock on the range between
RangeS-S 14 keys; shared lock on the key at the
end of the range
Shared lock on the range between
RangeS-U 15 keys; update lock on the key at the
end of the range
Exclusive lock to prevent inserts on
RangeIn-Null 16 the range between keys; no lock on
the keys themselves
Exclusive lock on the range between
RangeX-X 22 keys; exclusive lock on the key at the
end of the range
Conversion lock created by S and
RangeIn-S 17
RangeIn_Null lock
Conversion lock created by U and
RangeIn-U 18
RangeIn_Null lock
Conversion of X and RangeIn_Null
RangeIn-X 19
lock
Conversion of RangeIn_Null and
RangeIn-U 20
RangeS_S lock
Conversion of RangeIn_Null and
RangeX-U 21
RangeS_U lock

The RangeIn-Null lock is acquired when SQL Server attempts to insert into the range between keys. This type of
lock is not often seen because it is typically very transient. It is held only until the correct location for insertion is found,
and then the lock is escalated into an X lock. However, if you have one transaction scan a range of data using the
Serializable isolation level and then another transaction tries to insert into that range, the second transaction will have a
lock request in a WAIT state with the RangeIn-Null mode. You can observe this using the sp_lock command, as I'll
describe in the section on viewing locks later in the chapter.

Lock Granularity
SQL Server can lock user data resources (not system resources, which are protected with latches) at the table, page,
or row level. SQL Server also locks index keys and ranges of index keys. Figure 14-3 shows the possible lock levels
in a table. Note that if the table has a clustered index, the data rows are at the leaf level of the clustered index and they
are locked with key locks instead of row locks.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 14-3. Levels of granularity for SQL Server locks.

The syslockinfo table keeps track of each lock by storing the type of resource locked (such as a row, key, or page),
the mode of the lock, and an identifier for the specific resource. When a process requests a lock, SQL Server
compares the lock requested to the resources already listed in the syslockinfo table and looks for an exact match on
the resource type and identifier. (The lock modes don't have to be the same to yield an exact match.) However, if one
process has a row exclusively locked in the authors table, for example, another process might try to get a lock on the
entire authors table. Since these are two different resources, SQL Server does not find an exact match unless
additional information is already stored in syslockinfo. This is what intent locks are for. The process that has the
exclusive lock on a row of the authors table also has an intent exclusive lock on the page containing the row and
another intent exclusive lock on the table containing the row. When the second process attempts to acquire the
exclusive lock on the table, it finds a conflicting row already in the syslockinfo table on the same lock resource (the
authors table). Not all requests for locks on resources that are already locked will result in a conflict. A conflict
occurs when one process requests a lock on a resource that is already locked by another process in an incompatible
lock mode. For example, two processes can each acquire shared locks on the same resource because shared locks
are compatible with each other. I'll discuss lock compatibility in detail later in this chapter.

Key Locks
SQL Server 2000 supports two kinds of key locks, whose use depends on the isolation level of the current
transaction. If the isolation level is Read Committed or Repeatable Read, SQL Server tries to lock the actual index
keys accessed while processing the query. With a table that has a clustered index, the data rows are the leaf level of
the index, and you will see key locks acquired. If the table is a heap, you might see key locks for the nonclustered
indexes and row locks for the actual data.

If the isolation level is Serializable, the situation is special. We want to prevent phantoms, which means that if we have
scanned a range of data within a transaction, we need to lock enough of the table to make sure that no one can insert
a value into the range that was scanned. For example, we can issue the following query within an explicit transaction:
BEGIN TRAN
SELECT * FROM employees
WHERE salary BETWEEN 30000 AND 50000

Locks must be acquired to make sure that no new rows with salary values between 30000 and 50000 are inserted
before the end of the transaction. Prior to version 7.0, SQL Server guaranteed this by locking whole pages or even
the entire table. In many cases, however, this was too restrictive—more data was locked than the actual WHERE
clause indicated, resulting in unnecessary contention. SQL Server 2000 uses key-range locks, which are associated
with a particular key value in an index and indicate that all values between that key and the previous one in the index
are locked.

Suppose we have an index on the lastname field in the employees table. We are in TRANSACTION ISOLATION
LEVEL SERIALIZABLE and we issue this SELECT statement:
SELECT *
FROM employees
WHERE last_name BETWEEN 'Delaney' AND 'DuLaney'

If Dallas, Donovan, and Duluth are sequential leaf-level index keys in the table, the second two of these (Donovan
and Duluth) acquire key-range locks (although only one row, for Donovan, is returned in the result set). The
key-range locks prevent any inserts into the ranges ending with the two key-range locks. No values greater than
Dallas and less than or equal to Donovan can be inserted, and no values greater than Donovan and less than or
equal to Duluth can be inserted. Note that the key-range locks imply an open interval starting at the previous
sequential key and a closed interval ending at the key on which the lock is placed. These two key-range locks prevent
This document is created with the unregistered version of CHM2PDF Pilot

anyone from inserting either Delany or Delanie, which are in the range specified in the WHERE clause. However, the
key-range locks would also prevent anyone from inserting DeLancey (which is greater than Dallas and less than
Donovan) even though DeLancey is not in the query's specified range. Key-range locks are not perfect, but they do
provide much greater concurrency than locking whole pages or tables, which was the only possibility in previous SQL
Server versions.

Additional Lock Resources

Locking is also done on extents—units of disk space that are 64 KB in size (eight pages of 8 KB each). This kind of
locking occurs automatically when a table or an index needs to grow and a new extent must be allocated. You can
think of an extent lock as another type of special purpose latch, but it does show up in the output of the sp_lock
procedure. Extents can have both shared extent and exclusive extent locks.

When you examine the output of sp_lock, notice that most processes hold a lock on at least one database. In fact,
any process holding locks in any database other than master or tempdb will have a DB lock for that database. These
are always shared locks and are used by SQL Server for determining when a database is in use. SQL Server detects
DB locks when determining whether a database can be dropped, restored, or closed. Since master and tempdb
cannot be dropped or closed, DB locks are unnecessary. In addition, tempdb is never restored, and to restore the
master database the entire server must be started in single-user mode, so again, DB locks are unnecessary.
Generally, you don't need to be concerned with extent or database locks, but you might see them if you are running
sp_lock or perusing syslockinfo.

Application Locks
The method used by SQL Server to store information about locking and to check for incompatible locks is very
straightforward and extensible. As you've seen, the SQL Server lock manager knows nothing about the object it is
locking. It works only with strings representing the resources without knowing the actual structure of the item. If two
processes are trying to obtain incompatible locks on the same resource, blocking will occur.

If the SQL Server developers were to decide to allow you to lock individual columns as well as rows, pages, and
tables, they could simply decide on an internal code number for column locks, and then we could add that to the list of
resources in Table 14-3.

Instead of adding new lock resources, SQL Server 2000 lets you extend the resources that can be locked. You can
take advantage of the supplied mechanisms for detecting blocking and deadlocking situations, and you can choose to
lock anything you like. These lock resources are called application locks. To define an application lock, you specify
a name for the resource you are locking, a mode, an owner, and a timeout.

Two resources are considered to be the same resource and are subject to blocking if they have the same name and
the same owner in the same database. Remember that by lock owner we mean the session, the transaction, or a
cursor. For your own application locks, the only possible owners are transaction and session. Two requests for locks
on the same resource can be granted if the modes of the locks requested are compatible. The locks are checked for
compatibility using the same compatibility matrix used for SQL Server supplied locks.

For example, suppose you have a stored procedure that only one user at a time should execute. You can "lock" that
procedure by using the sp_getapplock procedure to acquire a special lock, which means that someone is using this
procedure. When the procedure is complete, you can use sp_releaseapplock to release the lock:
EXEC sp_getapplock 'ProcLock', 'Exclusive', 'session'
EXEC MySpecialProc <parameter list>
EXEC sp_releaseapplock 'ProcLock', 'session'

Until the lock is released using sp_releaseapplock, or until the session terminates, no other session can execute this
procedure as long as it follows the protocol and uses sp_getapplock to request rights to the procedure before trying
This document is created with the unregistered version of CHM2PDF Pilot

to execute it. SQL Server doesn't know what the resource ProcLock means. It just adds a row to the syslockinfo
table that it will use to compare against other requested locks. Note that the procedure itself is not really locked. If
another user or application doesn't know that this is a special procedure and tries to execute MySpecialProc without
acquiring the application lock, SQL Server will not prevent the session from executing the procedure.

The resource name used in these procedures can be any identifier up to 255 characters long. The possible modes of
the lock, which is used to check compatibility with other requests for this same resource, are Shared, Update,
Exclusive, IntentExclusive, and IntentShared. There is no default; you must specify a mode. The possible values for
lock owner, the third parameter, are transaction (the default) or session. A lock with an owner of transaction must be
acquired with a user-defined transaction, and it will be automatically released at the end of the transaction without any
need to call sp_releaseapplock. A lock with an owner of session will be released automatically only when the session
disconnects.

Identifying Lock Resources

When the lock manager tries to determine whether a requested lock can be granted, it checks the syslockinfo table
to determine whether a matching lock with a conflicting lock mode already exists. It compares locks by looking at the
database ID (dbid), the object ID (objid), the type of resource locked, and the description of the specific resource
referenced by the lock. The lock manager knows nothing about the meaning of the resource description. It simply
compares the strings identifying the lock resources to look for a match. If it finds a match, it knows the resource is
already locked; it then uses the lock compatibility matrix to determine whether the current lock is compatible with the
one being requested. Table 14-3 shows all the lock resources, the abbreviations used in the output of sp_lock, and
the information used to define the actual resource locked.

Table 14-3. Lockable resources in SQL Server.

Internal Code for Resource
Resource Abbreviation Example
Resource Description
None; the database is
always indicated in
Database DB 2 the dbid column for
every locked
resource.
261575970 (Note
that sp_lock shows
the table ID in its own
Table TAB 5 The table ID.
column rather than in
the resource
description column.)
File number:page
Extent EXT 8 number of the first 1:96
page of the extent.
File number:page
Page PAG 6 number of the actual 1:104
table or index page.
This document is created with the unregistered version of CHM2PDF Pilot

A hashed value
derived from all the
key components and
the locator. For a
nonclustered index on
Index Key KEY 7 a heap, where ac0001a10a00
columns c1 and c2are
indexed, the hash
would contain
contributions from c1,
c2, and the RID.
Index Key Range KEY 7 Same as Index Key ac0001a10a00
File number:page
Row RID 9 number:slot number of 1:161:3
the actual row.
A hashed value
Application APP 10 derived from the name MyPr8adaea5f
given to the lock.

Note that key locks and key-range locks have identical resource descriptions. When we look at some examples of
output from the sp_lock procedure, you'll see that you can distinguish between these types of locks by the value in the
lock mode column.

Lock Duration
The length of time that a lock is held depends primarily on the mode of the lock and the transaction isolation level in
effect. The default isolation level for SQL Server is Read Committed. At this level, shared locks are released as soon
as SQL Server has read and processed the locked data. An exclusive lock is held until the end of the transaction,
whether it is committed or rolled back. An update lock is also held until the end of the transaction unless it has been
promoted to an exclusive lock, in which case the exclusive lock, as with all exclusive locks, remains for the duration of
the transaction. If your transaction isolation level is Repeatable Read or Serializable, shared locks have the same
duration as exclusive locks. That is, they are not released until the transaction is over.

In addition to changing your transaction isolation level, you can control the lock duration by using query hints. I'll
discuss query hints for locking and for other purposes in Chapter 16.

Lock Ownership
Lock duration can also be affected by the lock ownership. There are three types of lock owners: transactions,
cursors, and sessions. These are available through the req_ownertype column in the syslockinfo table. (This
information is not visible through the sp_lock stored procedure.) A req_ownertype value of 1 indicates that the lock
is owned by transaction, and its duration is as discussed as described in the previous section. Most of our locking
discussion, in fact, deals with locks owned by a transaction.

A cursor lock has a req_ownertype value of 2. If a cursor is opened using a locking mode of SCROLL_LOCKS, a
cursor lock is held on every row fetched until the next row is fetched or the cursor is closed. Even if the transaction
commits before the next fetch, the cursor lock is not released.

Locks owned by a session have a req_ownertype value of 3. A session lock is one taken on behalf of a process that
is outside the scope of a transaction. The most common example is a database lock, as discussed earlier. A process
acquires a session lock on the database when it issues the USE database command, and that lock isn't released until
another USE command is issued or until the process is disconnected.
This document is created with the unregistered version of CHM2PDF Pilot

Viewing Locks
To see the locks currently outstanding in the system as well as those that are being waited for, examine the
syslockinfo system table or execute the system stored procedure sp_lock. The syslockinfo table is not really a
system table. It is not maintained on disk because locks are not maintained on disk. Rather, it is materialized in table
format based on the lock manager's current accounting of locks each time syslockinfo is queried. Another way to
watch locking activity is with the excellent graphical representation of locking status provided by SQL Server
Enterprise Manager. Even those who think that GUIs are for wimps can appreciate SQL Server Enterprise
Manager's view of locking.

In some cases, the output of sp_lock can be voluminous. You can reduce the output by specifying one or two
process ID values; sp_lock will then show you only locks held by those processes. The process ID for a particular
connection is available using the system function @@spid. You can execute sp_lock and specify only your current
connection:
EXEC sp_lock @@spid

However, even limiting the output to just the locks for the current connection can sometimes generate more output
that you're interested in. To produce the lock output, SQL Server must translate internal ID numbers for the type of
lock and mode of lock into the strings shown in Tables 14-1, 14-2, and 14-3. To do this translation, SQL Server
uses the spt_values table in the master database as a giant lookup table. If you're using the Serializable isolation
level, locks can be held on this table in the master database as well as on temporary tables in tempdb. Having to
wade through these additional locks—which exist only because you're running sp_lock to examine your locks—can
make it difficult to understand the locks on your user data. To help solve this problem, I have written a modified
sp_lock procedure called sp_lock2, which does not print out any locks in the master, model, tempdb, or msdb
databases. In addition, the procedure translates the database ID into the database name. You can find the script to
create sp_lock2 on the companion CD.

The following examples show what each of the lock types and modes discussed earlier look like when reported by
the sp_lock2 procedure. Note that the call to the sp_lock2 procedure is preceded by the keyword EXECUTE,
which is required when the call to a stored procedure is not the first item in a batch. Note also that the sp_lock2
procedure is given an argument of @@spid so that we'll see only locks for the current process.

Example 1: SELECT with Default Isolation Level

SQL BATCH
USE PUBS
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
BEGIN TRAN
SELECT * FROM authors
WHERE au_lname = 'Ringer'
EXEC sp_lock2 @@spid
COMMIT TRAN

OUTPUT OF sp_lock2
spid Database ObjId IndId Type Resource Mode Status
------ --------- ------ ------ ---- --------------- -------- ------
52 pubs 0 0 DB S GRANT

The only lock we have is the DB lock. No locks on the authors table are held at this point because the batch was
This document is created with the unregistered version of CHM2PDF Pilot

doing only select operations that acquired shared locks. By default, the shared locks are released as soon as the data
has been read, so by the time sp_lock2 is executed, the locks are no longer held.

Example 2: SELECT with Repeatable Read Isolation Level

SQL BATCH
USE PUBS
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRAN
SELECT * FROM authors
WHERE au_lname = 'Ringer'
EXEC sp_lock2 @@spid
COMMIT TRAN

OUTPUT OF sp_lock2
spid Database ObjId IndId Type Resource Mode Status
----- --------- ----------- ------ ---- ---------------- ----- ------
54 pubs 0 0 DB S GRANT
54 pubs 1977058079 2 KEY (62039d7395e8) S GRANT
54 pubs 1977058079 1 PAG 1:88 IS GRANT
54 pubs 1977058079 2 PAG 1:116 IS GRANT
54 pubs 1977058079 1 KEY (04015bb61919) S GRANT
54 pubs 1977058079 1 KEY (1201b4159b48) S GRANT
54 pubs 1977058079 2 KEY (6e021955d8e9) S GRANT
54 pubs 1977058079 0 TAB IS GRANT

Because the authors table has a clustered index, the rows of data are all index rows in the leaf level. The locks on
the individual rows are marked as key locks instead of row locks. There are also key locks at the leaf level of the
nonclustered index on the table. In the authors table, the nonclustered index is on the author's last name (au_lname)
column, and that is the index being traversed to find the specified rows. You can tell the clustered and nonclustered
indexes apart by the value in the IndId field: the data rows have an IndId value of 1, and the nonclustered index rows
have an IndId value of 2. Because the transaction isolation level is Repeatable Read, the shared locks are held until
the transaction is finished. Note that the two rows and two index rows have shared (S) locks, and the data and index
pages, as well as the table itself, have intent shared (IS) locks.

Example 3: SELECT with Serializable Isolation Level

SQL BATCH
USE PUBS
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRAN
SELECT * FROM authors
WHERE au_lname = 'Ringer'
EXEC sp_lock2 @@spid
COMMIT TRAN

OUTPUT OF sp_lock2
spid Database ObjId IndId Type Resource Mode Status
----- -------- ----------- ----- ---- --------------- -------- ------
56 pubs 0 0 DB S GRANT
56 pubs 1977058079 2 KEY (62039d7395e8) RangeS-S GRANT
56 pubs 1977058079 1 PAG 1:88 IS GRANT
56 pubs 1977058079 2 PAG 1:116 IS GRANT
This document is created with the unregistered version of CHM2PDF Pilot

56 pubs 1977058079 1 KEY (04015bb61919) S GRANT

56 pubs 1977058079 1 KEY (1201b4159b48) S GRANT
56 pubs 1977058079 2 KEY (6e021955d8e9) RangeS-S GRANT
56 pubs 1977058079 0 TAB IS GRANT
56 pubs 1977058079 2 KEY (4903312f82bf) RangeS-S GRANT

The locks held with the Serializable isolation level are almost identical to those held with the Repeatable Read
isolation level. The main difference is in the mode of the lock. The two-part mode RangeS-S indicates a key-range
lock in addition to the lock on the key itself. The first part (RangeS) is the lock on the range of keys between (and
including) the key holding the lock and the previous key in the index. The key-range locks prevent other transactions
from inserting new rows into the table that meet the condition of this query; that is, no new rows with a last name of
Ringer can be inserted. The key-range locks are held on ranges in the nonclustered index on au_lname and
au_fname (IndId = 2) because that is the index used to find the qualifying rows. There are three key locks in the
nonclustered index because three different ranges need to be locked. SQL Server must lock the range from the key
preceding the first Ringer in the index up to the first Ringer, it must lock the range between the two instances of
Ringer, and it must lock the range from the second Ringer to the next key in the index. (So actually nothing between
Ringer and the previous key, Panteley, and nothing between Ringer and the next key, Smith, could be inserted. For
example, we could not insert an author with the last name Pike or Singh.)

Example 4: Update Operations

SQL BATCH
USE PUBS
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
BEGIN TRAN
UPDATE authors
SET contract = 0
WHERE au_lname = 'Ringer'
EXEC sp_lock2 @@spid
COMMIT TRAN

OUTPUT OF sp_lock2
spid Database ObjId IndId Type Resource Mode Status
----- --------- ----------- ------ ---- --------------- ----- ------
57 pubs 0 0 DB S GRANT
57 pubs 1977058079 1 PAG 1:88 IX GRANT
57 pubs 1977058079 1 KEY (04015bb61919) X GRANT
57 pubs 1977058079 1 KEY (1201b4159b48) X GRANT
57 pubs 1977058079 0 TAB IX GRANT

The two rows in the leaf level of the clustered index are locked with X locks. The page and the table are then locked
with IX locks. I mentioned earlier that SQL Server actually acquires update locks while it looks for the rows to
update. However, these are escalated to X locks when the actual update is done, and by the time the sp_lock2
procedure is run, the update locks are gone. Unless you actually force update locks with a query hint, you might never
see them in the output of sp_lock or sp_lock2.

Example 5: Update with Serializable Isolation Level Using an Index

SQL BATCH
USE PUBS
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRAN
UPDATE authors
This document is created with the unregistered version of CHM2PDF Pilot

SET contract = 0
WHERE au_lname = 'Ringer'
EXEC sp_lock2 @@spid
COMMIT TRAN

OUTPUT OF sp_lock2
spid Database ObjId IndId Type Resource Mode Status
----- --------- ----------- ----- ---- --------------- -------- ------
53 pubs 0 0 DB S GRANT
53 pubs 1977058079 2 KEY (62039d7395e8) RangeS-U GRANT
53 pubs 1977058079 0 TAB IX GRANT
53 pubs 1977058079 2 KEY (4903312f82bf) RangeS-U GRANT
53 pubs 1977058079 1 PAG 1:88 IX GRANT
53 pubs 1977058079 2 KEY (6e021955d8e9) RangeS-U GRANT
53 pubs 1977058079 2 PAG 1:116 IU GRANT
53 pubs 1977058079 1 KEY (04015bb61919) X GRANT
53 pubs 1977058079 1 KEY (1201b4159b48) X GRANT

Again, notice that the key-range locks are on the nonclustered index used to find the relevant rows. The range
interval itself needs only a share lock to prevent insertions, but the searched keys have U locks so that no other
process can attempt to update them. The keys in the table itself (IndId = 1) obtain the exclusive lock when the actual
modification is made.

Now let's look at an update operation with the same isolation level when no index can be used for the search.

Example 6: Update with Serializable Isolation Level Not Using an

Index
SQL BATCH
USE PUBS
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRAN
UPDATE authors
SET contract = 1
WHERE state = 'UT'
EXEC sp_lock2 @@spid
COMMIT TRAN

OUTPUT OF sp_lock2
spid Database ObjId IndId Type Resource Mode Status
---- -------- ----------- ------ ---- ---------------- -------- ------
53 pubs 1977058079 1 KEY (0b018636f9dc) RangeS-U GRANT
53 pubs 0 0 DB S GRANT
53 pubs 1977058079 1 KEY (100125852812) RangeS-U GRANT
53 pubs 1977058079 1 KEY (1001898f02b5) RangeS-U GRANT
53 pubs 1977058079 1 KEY (02010b688383) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0401c963b692) RangeS-U GRANT
53 pubs 1977058079 1 KEY (06013d38d450) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0d01ac7ad21d) RangeS-U GRANT
53 pubs 1977058079 0 TAB IX GRANT
53 pubs 1977058079 1 KEY (0c01f170d106) RangeS-U GRANT
53 pubs 1977058079 1 KEY (10018e6baefb) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0501dc9d152f) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0801c4f7a625) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0c019fc272ba) RangeS-U GRANT
This document is created with the unregistered version of CHM2PDF Pilot

53 pubs 1977058079 1 KEY (0101aedb232b) RangeS-U GRANT

53 pubs 1977058079 1 KEY (0601989a35e6) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0e01733a43ca) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0601d808a263) RangeS-U GRANT
53 pubs 1977058079 1 KEY (070199a50c2d) RangeS-U GRANT
53 pubs 1977058079 1 PAG 1:88 IX GRANT
53 pubs 1977058079 1 KEY (100198c3f985) RangeS-U GRANT
53 pubs 1977058079 1 KEY (ffffffffffff) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0701a9d28094) RangeS-U GRANT
53 pubs 1977058079 1 KEY (02014f0bec4e) RangeS-U GRANT
53 pubs 1977058079 1 KEY (0901ce1ba5f0) RangeS-U GRANT
53 pubs 1977058079 1 KEY (04015bb61919) RangeX-X GRANT
53 pubs 1977058079 1 KEY (1201b4159b48) RangeX-X GRANT

The locks here are similar to those in the previous example except that all the locks are on the table itself (IndId = 1).
A clustered index scan (on the entire table) had to be done, so all keys initially received the RangeS-U lock, and
when the two row were eventually modified, the locks on those keys escalated to the RangeX-X lock.

Example 7: Creating a Table

SQL BATCH
USE PUBS
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
BEGIN TRAN
SELECT *
INTO newTitles
FROM titles
WHERE price < 5
EXEC sp_lock2 @@spid
COMMIT TRAN

OUTPUT OF sp_lock2
spid Database ObjId IndId Type Resource Mode Status
---- --------- ----------- ------ ---- ---------------- ----- ------
57 pubs 0 0 DB [BULK-OP-LOG] NULL GRANT
57 pubs 0 0 DB S GRANT
57 pubs 0 0 DB [BULK-OP-DB] NULL GRANT
57 pubs 1 0 TAB IX GRANT
57 pubs 3 0 TAB IX GRANT
57 pubs 2 0 TAB IX GRANT
57 pubs 3 2 KEY (e40194df4eba) X GRANT
57 pubs 3 2 KEY (7b0152f4ac0f) X GRANT
57 pubs 3 2 KEY (ec01c5f30203) X GRANT
57 pubs 0 0 PAG 1:146 X GRANT
57 pubs 0 0 PAG 1:145 X GRANT
57 pubs 3 2 KEY (410288dc0a1c) X GRANT
57 pubs 0 0 PAG 1:199 X GRANT
57 pubs 0 0 PAG 1:198 X GRANT
57 pubs 0 0 PAG 1:197 X GRANT
57 pubs 0 0 PAG 1:196 X GRANT
57 pubs 0 0 PAG 1:195 X GRANT
57 pubs 0 0 PAG 1:194 X GRANT
57 pubs 0 0 PAG 1:193 X GRANT
57 pubs 0 0 PAG 1:192 X GRANT
57 pubs 0 0 EXT 1:192 X GRANT
57 pubs 3 1 KEY (a1007beae6ee) X GRANT
57 pubs 3 1 KEY (9a00f1a5ee93) X GRANT
57 pubs 3 1 KEY (a000954553fc) X GRANT
57 pubs 3 1 KEY (9d00c35530a4) X GRANT
This document is created with the unregistered version of CHM2PDF Pilot

57 pubs 3 1 KEY (9c002dfa85b6) X GRANT

57 pubs 3 2 KEY (4d02b937e929) X GRANT
57 pubs 3 2 KEY (c6029c7b5ab8) X GRANT
57 pubs 1 1 KEY (97009fa6c549) X GRANT
57 pubs 3 2 KEY (d2016d108bf1) X GRANT
57 pubs 3 1 KEY (98007a6de739) X GRANT
57 pubs 3 1 KEY (990094c2522b) X GRANT
57 pubs 3 1 KEY (9e00a6328c1c) X GRANT
57 pubs 3 1 KEY (9b00489d390e) X GRANT
57 pubs 3 1 KEY (9f00f022ef44) X GRANT
57 pubs 3 2 KEY (d501db08ee70) X GRANT
57 pubs 3 2 KEY (2f02d5933c81) X GRANT
57 pubs 0 0 IDX IDX: 5:725577623 X GRANT
57 pubs 725577623 0 TAB Sch-M GRANT
57 pubs 1 2 KEY (2e02001f6a2c) X GRANT
57 pubs 2 1 KEY (9700cdd96cc9) X GRANT
57 pubs 3 2 KEY (6702012f70a8) X GRANT
57 pubs 1 3 KEY (9700eaa6a30d) X GRANT

Very few of these locks are actually acquired on elements of the new table. In the ObjId column, notice that most of
the objects have an ID of less than 100, which means that they are system tables. As the new newTitles table is built,
SQL Server acquires locks on sysobjects and syscolumns to record information about this new table. Also notice the
schema modification (Sch-M) lock on the new table as well as the extent (EXT) locks. While the table is being built,
the extents are not marked as belonging to the table; you can see that the ObjId is 0. In the output in this example, the
extent ID is shown as 1:192. This means that page 192 in file 1 is the first page of the extent. You can also see that
the subsequent seven pages (193-199) in this extent are all exclusively locked while the table is being created.

Example 8: Row Locks

SQL BATCH
USE PUBS
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
BEGIN TRAN
UPDATE newTitles
SET price = 3.99
WHERE type = 'business'
EXEC sp_lock2 @@spid
ROLLBACK TRAN

OUTPUT OF sp_lock2
spid Database ObjId IndId Type Resource Mode Status
---- --------- ----------- ------ ---- ---------- -------- ------
58 pubs 0 0 DB S GRANT
58 pubs 693577509 0 TAB IX GRANT
58 pubs 693577509 0 PAG 1:192 IX GRANT
58 pubs 693577509 0 RID 1:192:0 X GRANT

There are no indexes on the newTitles table, so the lock on the actual row meeting our criterion is an exclusive (X)
lock on the row (RID). As expected, IX locks are taken on the page and the table.
This document is created with the unregistered version of CHM2PDF Pilot

34

Lock Compatibility
Two locks are compatible if one lock can be granted while another lock on the same object held by a different
process is outstanding. On the other hand, if a lock requested for an object is not compatible with a lock currently
being held, the requesting connection must wait for the lock. For example, if a shared page lock exists on a page,
another process requesting a shared page lock for the same page is granted the lock because the two lock types are
compatible. But a process that requests an exclusive lock for the same page is not granted the lock because an
exclusive lock is not compatible with the shared lock already held. Table 14-4 summarizes the compatibility of locks
in SQL Server. Yes indicates that the lock held is compatible with the lock requested; No means that they're not
compatible.

Table 14-4. SQL Server's lock compatibility matrix.

Request Existing Grant Mode
ed Mode IS S U IX SIX X Sch-S Sch-M BU
IS Yes Yes Yes Yes Yes No Yes No No
S Yes Yes Yes No No No Yes No No
U Yes Yes No No No No Yes No No
IX Yes No No Yes No No Yes No No
SIX Yes No No No No No Yes No No
X No No No No No No Yes No No
Sch-S Yes Yes Yes Yes Yes Yes Yes No Yes
Sch-M No No No No No No No No No
BU No No No No No No Yes No Yes

Lock compatibility comes into play between locks on different resources, such as table locks and page locks. A table
and a page obviously represent an implicit hierarchy since a table is made up of multiple pages. If an exclusive page
lock is held on one page of a table, another process cannot get even a shared table lock for that table. This hierarchy
is protected using intent locks. A process acquiring an exclusive page lock, update page lock, or intent exclusive page
lock first acquires an intent exclusive lock on the table. This intent exclusive table lock prevents another process from
acquiring the shared table lock on that table. (Remember that intent exclusive and shared locks are not compatible.)

Similarly, a process acquiring a shared row lock must first acquire an intent shared lock for the table, which prevents
another process from acquiring an exclusive table lock. Or if the exclusive table lock already exists, the intent shared
lock is not granted and the shared page lock has to wait until the exclusive table lock is released. Without intent locks,
process A can lock a page in a table with an exclusive page lock and process B can place an exclusive table lock on
the same table and hence think that it has a right to modify the entire table, including the page that process A has
exclusively locked.
NOTE
At the risk of stating the obvious, lock compatibility is an issue only when the locks affect the same object. For
example, two or more processes can each hold exclusive page locks simultaneously as long as the locks are on
different pages or different tables.

Even if two locks are compatible, the requester of the second lock might still have to wait if an incompatible lock is
waiting. For example, suppose that process A holds a shared page lock. Process B requests an exclusive page lock
and must wait because the shared page lock and the exclusive page lock are not compatible. Process C requests a
shared page lock that is compatible with the shared page already outstanding to process A. However, the shared
page lock cannot be immediately granted. Process C must wait for its shared page lock because process B is ahead
This document is created with the unregistered version of CHM2PDF Pilot

of it in the lock queue with a request (exclusive page) that is not compatible.

By examining the compatibility of locks not only with processes granted but also processes waiting, SQL Server
prevents lock starvation, which can result when requests for shared locks keep overlapping so that the request for
the exclusive lock can never be granted.
This document is created with the unregistered version of CHM2PDF Pilot

34

Internal Locking Architecture

Locks are not on-disk structures—you won't find a lock field directly on a data page or a table header—because it
would be too slow to do disk I/O for locking operations. Locks are internal memory structures—they consume part
of the memory used for SQL Server. Each locked data resource (a row, index key, page, or table) requires 64 bytes
of memory to keep track of the database, the type of lock, and the information describing the locked resource. Each
process holding a lock also must have a lock owner block of 32 bytes. A single transaction can have multiple lock
owner blocks; a scrollable cursor sometimes uses several. Also, one lock can have many lock owner blocks, as in the
case with a shared lock. Finally, each process waiting for a lock has a lock waiter block of another 32 bytes. Since
lock owner blocks and lock waiter blocks have identical structures, I'll use the term lock owner block to refer to
both of them.

The lock manager maintains a lock hash table. Lock resources, contained within a lock block, are hashed to
determine a target hash slot in the hash table. (I'll discuss hashing in detail when I talk about the SQL Server hash join
algorithm in Chapter 15.) All lock blocks that hash to the same slot are chained together from one entry in the hash
table. Each lock block contains a 16-byte field that describes the locked resource. This 16-byte description is
viewable in the syslockinfo table in the rsc_bin column. I'll dissect that column later in this section. The lock block
also contains pointers to lists of lock owner blocks. A lock owner block represents a process that has been granted a
lock, is waiting for the lock, or is in the process of converting to the lock. Figure 14-4 shows the general lock
architecture.

Figure 14-4. SQL Server's locking architecture.

The number of slots in the hash table is based on the system's physical memory, as shown in Table 14-5. All
instances of SQL Server on the same machine will have a hash table with the same number of slots. Each entry in the
lock hash table is 16 bytes in size and consists of a pointer to a list of lock blocks and a spinlock to guarantee
nonconcurrent access to the same slot.

Table 14-5. Number of slots in the internal lock hash table.

Physical Memory (MB) Number of Slots Memory Used
< 32 2^13 = 8K 128 KB
This document is created with the unregistered version of CHM2PDF Pilot

>= 32 and < 64 2^15 = 32K 512 KB

>= 64 and < 128 2^16= 64K 1 MB
>= 128 and < 512 2^17 = 128K 2 MB
>= 512 and < 1024 2^18 = 256K 4 MB
>= 1024 2^19 = 512K 8 MB

The lock manager preallocates a number of lock blocks and lock owner blocks at server startup. If the number of
locks is fixed by sp_configure, it allocates that configured number of lock blocks and the same number of lock owner
blocks. If the number is not fixed (0 means auto-tune), it allocates 500 lock blocks on a SQL Server 2000, Desktop
Edition server and 2500 lock blocks for the other editions. It allocates twice as many (2 * # lock blocks) of the lock
owner blocks. At their maximum, the static allocations can't consume more than 25 percent of the committed buffer
pool size.
NOTE
In this context, I use the term process to refer to a SQL Server subtask. Every user connection is referred to as a
process, as are the checkpoint manager, the lazywriter, the log writer, and the lock monitor. But these are only
subtasks within SQL Server, not processes from the perspective of the operating system, which considers the entire
SQL Server engine to be a single process with multiple threads.

When a request for a lock is made and no free lock blocks remain, the lock manager dynamically allocates new lock
blocks instead of denying the lock request. The lock manager cooperates with the global memory manager to
negotiate for server allocated memory. Each lock contains a flag indicating whether the block was dynamically
allocated or preallocated. When necessary, the lock manager can free the dynamically allocated lock blocks. The
lock manager is limited to 60 percent of the Buffer Manager's committed target size allocation to lock blocks and lock
owner blocks.

Lock Blocks
The lock block is the key structure in SQL Server's locking architecture, as shown earlier in Figure 14-4. A lock
block contains the following information:
 Lock resource name
 Pointers to connect the lock blocks to the lock hash table
 General summary information
 Pointer to a list of lock owner blocks for locks on this resource that have been granted (granted list)
 Pointer to a list of lock owner blocks for locks on this resource that are waiting to be converted to another
lock mode (convert list)
 Pointer to a list of lock owner blocks for locks that have been requested on this resource but have not yet
been granted (wait list)

The lock resource block is the most important element of the lock block. Its structure is shown in Figure 14-5. Each
"row" in the figure represents 4 bytes, or 32 bits.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 14-5. The structure of a lock resource block.

The meanings of the fields shown in Figure 14-5 are described in Table 14-6. The value in the "resource type" byte is
one of the locking resources described earlier in Table 14-3. The meaning of the values in the three data fields varies
depending on the type of resource being described. SR indicates a subresource (which I'll describe shortly).

Table 14-6. The meaning of fields in the lock resource block.

Resource Contents
Resource Type
Data 1 Data 2 Data 3
Database (2) SR 0 0
File (3) File ID 0 0
Index (4) Object ID SR Index ID
Table (5) Object ID SR 0
Page (6) Page Number 0
Key (7)(For a Key
resource, Data2 is 2 bytes Object ID Index ID Hashed Key
and Data3 is 6 bytes)
Extent (8) Extent ID 0
RID (9) RID 0
Application (10) Application
(For an application resource
name resource, only the first
Application resource name
4 bytes of the name are
used; the remaining bytes
are hashed.)

Following are some of the possible SR (SubResouce) values:

 If the lock is on a DB resource, SR indicates one of the following:
 Full database lock
 Bulk operation lock
 If the lock is on a Table resource, SR indicates one of the following:
 Full table lock (default)
 Update statistics lock
 Compile lock
 If the lock is on an Index resource, SR indicates one of the following:
 Full index lock (default)
 Index ID lock
 Index name lock

Lock Owner Blocks

Each lock owned or waited for by a session is represented in a lock owner block. Lists of lock owner blocks form
the grant, convert, and wait lists that hang off of the lock blocks. Each lock owner block for a granted lock is linked
with all other lock owner blocks for the same transaction or session so that they can be freed as appropriate when the
transaction or session ends.

Syslockinfo Table
The system procedures sp_lock and sp_lock2 are based on information extracted from the syslockinfo table, which
This document is created with the unregistered version of CHM2PDF Pilot

exists only in the master database. Lots more information, including the name of the lock owner, is kept in the
syslockinfo table. Table 14-7 shows the columns in that table. Columns prefixed by rsc_ are taken from a lock
resource block. Columns prefixed by req_ are taken from a lock owner block.

Table 14-7. Columns in the syslockinfo table.

Column Name Description
Textual description of a lock resource. Contains a portion
rsc_text
of the resource name.
Binary lock resource. Described in Figure 14-5 and Table
rsc_bin
14-6.
Lock value block. Some resource types might include
additional data in the lock resource that is not hashed by
the lock manager to determine ownership of a particular
rsc_valblk lock resource. For example, page locks are not owned by
a particular object ID. For lock escalation and other
purposes, however, the object ID of a page lock might be
placed in the lock value block.
rsc_dbid Database ID.
rsc_indid Index ID associated with the resource, if appropriate.
rsc_objid Object ID associated with the resource, if appropriate.
rsc_type Resource type from Table 14-3.
rsc_flag Internal resource flags.
Lock request mode. This column is the lock mode of the
requester and represents either the granted mode or the
req_mode
convert or waiting mode. Can be any of the modes listed
in Tables 14-1 or 14-2.
Status of the lock request.

1 = Granted
req_status
2 = Converting

3 = Waiting
Lock reference count. Each time a transaction asks for a
lock on a particular resource, a reference count is
req_refcnt
incremented. The lock cannot be released until the
reference count equals 0.
req_cryrefcnt Reserved for future use. Always set to 0.
Lock lifetime bitmap. During certain query processing
strategies, locks must be maintained on resources until the
query processor has completed a particular phase of the
query. The lock lifetime bitmap is used by the query
req_lifetime processor and the transaction manager to denote groups
of locks that can be released when a certain phase of a
query is completed. Certain bits in the bitmap are used to
denote locks that are held until the end of a transaction,
even if their reference count equals 0.
req_spid Process ID of the session requesting the lock.
Execution context ID (ECID). Used to denote which
req_ecid
thread in a parallel operation owns a particular lock.
This document is created with the unregistered version of CHM2PDF Pilot

Type of object associated with the lock. Can be one of

the following:

1 = Transaction

2 = Cursor
req_ownertype
3 = Session

4 = ExSession
Note that 3 and 4 represent a special version of session
locks that track database and filegroup locks,
respectively.
Unique transaction ID used in syslockinfo and in SQL
req_transactionID
Profiler events.
Identifies the Unit of Work (UOW) ID of the DTC
req_transactionUOW transaction. For non-MS DTC transactions, UOW is set
to 0.

In syslockinfo, you can analyze the rsc_bin field as the resource block. Here's an example:
SELECT rsc_bin FROM master..syslockinfo

Here's an example result row:

0x00070500AAA1534E0100080000000000

From left to right, with the appropriate byte swapping within each field, this example means the following:
 Byte 0: Flag: 0x00
 Bytes 1: Resource Type: 0x07, a Key resource
 Bytes 2 through 3: DBID: 0x0005

And since it is a Key resource:

 Bytes 4 through 7: ObjectID: 0x4E53A1AA (1314103722 decimal)
 Bytes 8 through 9: IndexID: 0x0001 (the clustered index)
 Bytes 10 through 16: Hash Key Name: 0x080000000000
This document is created with the unregistered version of CHM2PDF Pilot

34

Bound Connections
Remember that the issue of lock contention applies only between different SQL Server processes. A process holding
locks on a resource does not lock itself from the resource—only other processes are denied access. But any other
process (or connection to SQL Server) can actually execute as the same application and user. It is common for
applications to have more than one connection to SQL Server. Every such connection is treated as an entirely
different SQL Server process, and by default no sharing of the "lock space" occurs between connections, even if they
belong to the same user and the same application.

However, two or more different connections can share a lock space and hence not lock each other out. This
capability is known as a bound connection. With a bound connection, the first connection asks SQL Server to give
out its bind token. The bind token is passed by the application (using a client-side global variable, shared memory, or
another method) for use in subsequent connections. The bind token acts as a "magic cookie" so that other connections
can share the lock space of the original connection. Locks held by bound connections do not lock each other. (The
sp_getbindtoken and sp_bindsession system stored procedures get and use the bind token.)

Bound connections are especially useful if you're writing an extended stored procedure, which is a function written in
your own DLL, and that extended stored procedure needs to call back into the database to do some work. Without a
bound connection, the extended stored procedure collides with its own calling process's locks. When multiple
processes share a lock space and a transaction space by using bound connections, a COMMIT or ROLLBACK
affects all the participating connections. If you are going to use bound connections for the purpose of passing the bind
token to an extended stored procedure to call back into the server, you must use a second parameter of the constant
1. If no parameter of 1 is passed, the token cannot be used in an extended stored procedure.

Here's an example of using bound connections between two different windows in SQL Query Analyzer. In SQL
Server 2000, you must be inside of a transaction in order to get a bind token. Since we don't have a controlling
application to declare and store the bind token in a client-side variable, we have to actually copy it from the first
session and paste it into the second. So, in your first query window, you execute this batch:
DECLARE @token varchar(255)
BEGIN TRAN
EXEC sp_getbindtoken @token OUTPUT
SELECT @token
GO

This should return something like the following:

-----------dPe---5---.?j0U<_WP?1HMK-3/D8;@1

Normally, you wouldn't have to look at this messy string; your application would just store it and pass it on without
your ever having to see it. But for a quick example using SQL Query Analyzer, it's necessary to actually see the value.
You use your keyboard or mouse to select the token string that you received and use it in the following batch in a
second SQL Query Analyzer window:
EXEC sp_bindsession 'dPe---5---.?j0U<_WP?1HMK-3/D8;@1'
GO

Now go back to the first query window and execute a command that locks some data. Remember that we have
already begun a transaction in order to call sp_getbindtoken. You can use something like this:
This document is created with the unregistered version of CHM2PDF Pilot

USE pubs
UPDATE titles
SET price = $100
GO

This should exclusively lock every row in the titles table. Now go to the second query window and select from the
locked table:
SELECT title_id, price FROM titles
GO

You should be able to see all the $100 prices in the titles table, just as if you were part of the same connection as the
first query. Besides sharing lock space, the bound connection also shares transaction space. You can execute a
ROLLBACK TRAN in the second window even though the first one began the transaction. If the first connection
tries to then issue a ROLLBACK TRAN, it gets this message:
Server: Msg 3903, Level 16, State 1, Line 1
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION.
The transaction active in this session has been committed or aborted by
another session.

SQL Server keeps track of bound connections in an internal structure called the XCB (transaction control block),
which is used to relate multiple connections in a bound session. You can actually see which connections are bound
using SQL Query Analyzer and an undocumented DBCC command. First, you need the spid (Server Process ID) of
any connections you're interested in. You can get this value using the function @@spid:
SELECT @@spid

The DBCC command is called DBCC PSS. Most of the information returned is meaningful and relevant only to
Microsoft Product Support Services (PSS) engineers. It is really just a coincidence that Microsoft's support service is
called PSS; in this case, PSS stands for Process Status Structure. As you saw earlier with DBCC PAGE, for most of
the undocumented stored procedures you must first enable them using trace flag 3604. So suppose we've determined
that the Process ID value of the first connection above, which is updating the prices in the titles table, is 51, and the
connection we've bound to it is 54. The following command will print the PSS from both connections:
DBCC TRACEON(3604)
GO
DBCC PSS(1,51)
DBCC PSS(1,54)

Here's some partial output:

PSS:
----

PSS @0x193731D8
---------------
pspid = 51 m_dwLoginFlags = 0x03e0 plsid = 422666344
pbackground = 0
pbSid
-----
01 .
sSecFlags = 0x12 pdeadlockpri = 0 poffsets = 0x0
pss_stats = 0x0 ptickcount = 16806736
pcputickcount = 3363174526155 ploginstamp = 17
This document is created with the unregistered version of CHM2PDF Pilot

ptimestamp = 2000-08-13 12:31:47.793 prowcount = 21

plangid = 0 pdateformat = 1 pdatefirst = 7
Language = us_english RemServer = UserName = sa
poptions = 0x28480860 poptions2 = 0x3f438 pline = 1
pcurstepno = 0 prowcount = 18 pstatlist = 0
pcurcmd = 253 pseqstat = 0 ptextsize = 64512
pretstat = 0 pslastbatchstarttime = 2000-08-13 13:13:03.423
pmemusage = 19 hLicense = 0 tpFlags = 0x1
isolation_level = 2 fips_flag = 0x0 sSaveSecFlags = 0x0
psavedb = 0 pfetchstat = 0 pcrsrows = 0
pslastbatchtime = 2000-08-13 13:13:03.473 pubexecdb = 0
fInReplicatedProcExec = 0 pMsqlXact = 0x19373f88 presSemCount = [0]9951900
presSemCount = [0]9951900 pcputot = 180 pcputotstart = 170
pcpucmdstart = 170 pbufread = 32 pbufreadstart = 31
plogbufread = 533 plogbufreadstart = 428 pbufwrite = 5
pbufwritestart = 4 pLockTimeout = 4294967295 pUtilResultSet = 0x0

PSS:
----

PSS @0x195B51D8
---------------
pspid = 54 m_dwLoginFlags = 0x03e0 plsid = 422666344
pbackground = 0
pbSid
-----
01 .
sSecFlags = 0x12 pdeadlockpri = 0 poffsets = 0x0
pss_stats = 0x0 ptickcount = 18289118
pcputickcount = 3658807285722 ploginstamp = 24
ptimestamp = 2000-08-13 12:56:30.107 prowcount = 21
plangid = 0 pdateformat = 1 pdatefirst = 7
Language = us_english RemServer = UserName = sa
poptions = 0x28480860 poptions2 = 0x3f438 pline = 1
pcurstepno = 0 prowcount = 18 pstatlist = 0
pcurcmd = 253 pseqstat = 0 ptextsize = 64512
pretstat = 0 pslastbatchstarttime = 2000-08-13 13:13:56.380
pmemusage = 6 hLicense = 0 tpFlags = 0x1
isolation_level = 2 fips_flag = 0x0 sSaveSecFlags = 0x0
psavedb = 0 pfetchstat = 0 pcrsrows = 0
pslastbatchtime = 2000-08-13 13:13:56.420 pubexecdb = 0
fInReplicatedProcExec = 0 pMsqlXact = 0x195b5f88 presSemCount = [0]9951900
presSemCount = [0]9951900 pcputot = 40 pcputotstart = 30
pcpucmdstart = 30 pbufread = 3 pbufreadstart = 2
plogbufread = 56 plogbufreadstart = 33 pbufwrite = 0
pbufwritestart = 0 pLockTimeout = 4294967295 pUtilResultSet = 0x0

I won't even try to describe each of the values returned, mainly because I don't know them all. Some of them are
pretty self-explanatory, such as pslastbatchstarttime, Language, and UserName. You might be able to guess the
meaning of some of them. The value for the isolation level can be 0 through 4, with the numbers 1 through 4 mapping
to the four transaction isolation levels. The value of 2 in my output means Read Committed. You will sometimes get a
0 for this value, which means the default isolation, which is Read Committed for SQL Server.

I have shown the Process ID value and a value called plsid in boldface. The plsid value is the transaction space
identifier, although it doesn't look at all like the value returned in SQL Query Analyzer for the bind token. You can see
that the plsid value is the same for these two connections. If you run DBCC PSS and look at other connections
besides these two that have been bound together, you should notice that they all have different plsid values.
This document is created with the unregistered version of CHM2PDF Pilot

34

Row-Level vs. Page-Level Locking

The debate over whether row-level locking is better than page-level locking or vice-versa has been one of those
near-religious wars and warrants a few comments here. Although some people would have you believe that one is
always better, it's not really that simple.

Prior to version 7, the smallest unit of data that SQL Server could lock was a page. Even though many people
argued that this was unacceptable and it was impossible to maintain good concurrency while locking entire pages,
many large and powerful applications were written and deployed using only page-level locking. If they were well
designed and tuned, concurrency was not an issue, and some of these applications supported hundreds of active user
connections with acceptable response times and throughput. However, with the change in page size from 2 KB to 8
KB for SQL Server 7, the issue has become more critical. Locking an entire page means locking four times as much
data as in previous versions. Beginning with version 7, SQL Server implements full row-level locking, so any potential
problems due to lower concurrency with the larger page size should not be an issue. However, locking isn't free.
Considerable resources are required to manage locks. Recall that a lock is an in-memory structure of about 32 bytes,
with another 32 bytes for each process holding the lock and each process waiting for the lock. If you need a lock for
every row and you scan a million rows, you need more than 30 MB of RAM just to hold locks for that one process.

Beyond the memory consumption issues, locking is a fairly processing-intensive operation. Managing locks requires
substantial bookkeeping. Recall that, internally, SQL Server uses a lightweight mutex called a spinlock to guard
resources, and it uses latches—also lighter than full-blown locks—to protect non-leaf-level index pages. These
performance optimizations avoid the overhead of full locking. If a page of data contains 50 rows of data, all of which
will be used, it is obviously more efficient to issue and manage one lock on the page than to manage 50. That's the
obvious benefit of page locking—a reduction in the number of lock structures that must exist and be managed.

If two different processes each need to update a few separate rows of data and some of the rows needed by each
process happen to exist on the same page, one process must wait until the page locks of the other process are
released. If, in this case, you use row-level locking instead of page-level locking, the other process does not have to
wait. The finer granularity of the locks means that no conflict occurs in the first place because each process is
concerned with different rows. That's the obvious benefit of row-level locking. Which of these obvious benefits wins?
Well, the decision isn't clear cut, and it depends on the application and the data. Each type of locking can be shown to
be superior for different types of applications and usage.

The stored procedure sp_indexoption lets you manually control the unit of locking within an index. It also lets you
disallow page locks or row locks within an index. Since these options are available only for indexes, there is no way
to control the locking within the data pages of a heap. (But remember that if a table has a clustered index, the data
pages are part of the index and are affected by the sp_indexoption setting.) The index options are set for each table
or index individually. Two options, AllowRowLocks and AllowPageLocks, are both set to TRUE initially for every
table and index. If both of these options are set to FALSE for a table, only full table locks are allowed.

As mentioned earlier, SQL Server determines at runtime whether to initially lock rows, pages, or the entire table. The
locking of rows (or keys) is heavily favored. The type of locking chosen is based on the number of rows and pages to
be scanned, the number of rows on a page, the isolation level in effect, the update activity going on, the number of
users on the system needing memory for their own purposes, and so on.

Lock Escalation
SQL Server automatically escalates row, key, or page locks to coarser table locks as appropriate. This escalation
protects system resources—it prevents the system from using too much memory for keeping track of locks—and
This document is created with the unregistered version of CHM2PDF Pilot

increases efficiency. For example, after a query acquires many row locks, the lock level can be escalated to a table
lock. If every row in a table must be visited, it probably makes more sense to acquire and hold a single table lock than
to hold many row locks. A single table lock is acquired and the many row locks are released. This escalation to a
table lock reduces locking overhead and keeps the system from running out of locks. Because a finite amount of
memory is available for the lock structures, escalation is sometimes necessary to make sure the memory for locks
stays within reasonable limits.

When the lock count for one transaction exceeds 1250 or when the lock count for one index or table scan exceeds
765, the lock manager looks to see how much memory is being used for all locks in the system. If more than 40
percent of the memory pool is being used for locks, SQL Server attempts to escalate multiple page, key, or RID
locks into a table lock. SQL Server tries to find a table that is partially locked by the transaction and holds the largest
number of locks for which no escalation has already been performed, and which is capable of escalation. Multiple
RID, key, or page locks cannot be escalated to a table lock if some other processes hold incompatible locks on other
RIDs, keys, or pages of the same table. SQL Server will keep looking for other tables partially locked by the same
transaction until all possible escalations have taken place or the total memory used for locks drops under 40 percent.
Note that SQL Server never escalates to page locks; the result of a lock escalation is always a table lock.
This document is created with the unregistered version of CHM2PDF Pilot

34

Locking Hints and Trace Flags

Just as you can specify hints on queries to direct the query optimizer to choose a certain index or strategy in its query
plan, you can specify hints for locking. For example, if you know that your query will scan so many rows that its page
locks will escalate to table locks, you can direct the query to use table locks in the first place, which is more efficient.
We'll look at locking hints and locking contention issues in detail in Chapter 16.

For now, armed with the knowledge of locking you've gained here, you can observe the locking activity of your
system to understand how and when locks occur. Trace flag 1204 provides detailed information about deadlocks,
and this information can help you understand why locks are occurring and how to change the order of access to
objects to reduce them. Trace flag 1200 provides detailed locking information as every request for a lock is made.
(But its output is voluminous.) You'll see details about these trace flags in Chapter 16.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
SQL Server lets you manage multiple users simultaneously and ensure that transactions observe the properties of the
chosen isolation level. Locking guards data and the internal resources that make it possible for a multiple-user system
to operate like a single-user system. In this chapter, we looked at the locking mechanisms in SQL Server, including
full locking for data and leaf-level index pages and lightweight locking mechanisms for internally used resources.

We also looked at the types and modes of locks as well as lock compatibility and lock escalation. It is important to
understand the issues of lock compatibility if you want to design and implement high-concurrency applications. We
also looked at deadlocks and discussed ways to avoid them.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 15

The Query Processor

One of the most important components of the SQL Server engine is the query processor. Actually, there is no
specific component called the query processor; rather, several related components work closely together to process
the queries that you or your applications submit to the SQL Server engine for processing. Some of these components
are involved in parsing your queries and simplifying them in various ways. Others are involved in the actual execution
of your query.

Probably the most important and sophisticated of all the components involved in processing your queries is the query
optimizer. The optimizer's job is to produce an execution plan for your query or for each query in a batch or stored
procedure. The plan lists the steps to be carried out in order to execute your query and includes such information as
which index or indexes to use when accessing data from each table in the query. The plan also includes the strategy
for processing each JOIN operation, each GROUP BY operation, and each sort.

In this chapter, we'll look at the phases of query processing, paying particular attention to the query optimizer. I'll
show you the different ways that indexes can be used, and I'll describe strategies for processing the joining of tables
or the grouping of data. You'll also learn about the statistical information that SQL Server has available to determine
the optimum execution plan, as well as the various mechanisms that SQL Server can employ to reuse an execution
plan that was developed for a previous query.
This document is created with the unregistered version of CHM2PDF Pilot

34

The SQL Manager

The SQL manager, which we looked at in Chapter 3, is the driving force in much of SQL Server's query processing.
You might say that it's the heart of SQL Server itself. The SQL manager deals with all of the requests to run stored
procedures. It manages the procedure cache and the pseudo_system stored procedures, and it's involved in
autoparameterization of ad hoc queries, which I'll discuss shortly.

Typically, the SQL manager is invoked with an RPC message when you ask SQL Server to do some work for you.
However, when a SQL language statement comes in as an ad hoc SQL statement instead of as a procedure call, the
SQL manager is also involved. It can be involved when a procedure or a batch has an EXEC statement in it because
the EXEC is actually calling the SQL manager. If the SQL statement passes the autoparameterization template (which
I'll discuss below), the SQL manager is called to parameterize the queries. It's also called when ad hoc queries need
to be placed in the cache. The SQL manager really is the heart of SQL Server's query processing system.
This document is created with the unregistered version of CHM2PDF Pilot

34

Compilation and Optimization

It's important to be aware that compilation and execution are distinct phases of query processing and that the gap
between when SQL Server compiles a query and when the query is executed can be as short as a few microseconds
or as long as several days. Therefore, during compilation (which includes optimization), SQL Server has to determine
what kind of available information will be useful during the compilation. Not everything that is true at compilation time
will be true at execution time. For example, the optimizer might take into account how many concurrent users are
trying to access the same data that your query is trying to access. However, the number of concurrent users can
change dramatically from moment to moment. You really have to think of compilation and execution as two separate
activities, even when you're submitting an ad hoc SQL statement through SQL Query Analyzer and executing it
immediately.

When a query is ready to be processed by SQL Server, an execution plan for the query might already be available in
SQL Server's cache. If not, the query must be compiled. You'll see more details about how execution plans can be
saved and reused later in this chapter. The compilation process encompasses a few things. First, SQL Server goes
through the phases of parsing and normalization. Parsing is the process of dissecting and transforming your SQL
statements into compiler-ready data structures. Parsing also includes validation of the syntax to ensure that it is legal.

Parsing doesn't include things such as checking for valid table and column names, which are dealt with during
normalization. The normalization process basically determines the characteristics of the objects that you reference
inside your SQL statements and checks whether the semantics you're asking for make sense. For example, it is
semantically illogical to try to execute a table.

The next step is the compilation of the Transact-SQL code. The difference between Transact-SQL and SQL itself
can be confusing, and the developers at Microsoft use the two names as interchangeably as anyone else does.
However, there's actually an important difference. When I talk about processing the SQL statements, this includes all
of the DML statements: INSERT, UPDATE, DELETE, and SELECT. These are the only statements that can actually
be optimized. SQL Server also has a language that wraps around these DML statements, called Transact-SQL, or
TSQL. As you've seen in several previous chapters, Transact-SQL provides procedural constructs: IF statements,
WHILE statements, local variable declarations, and so forth. These are treated very differently inside the query
processor. The procedural logic of Transact-SQL is compiled by an engine that knows how to do procedural kinds
of things.

The SQL statements themselves are compiled by the query optimizer. The optimizer has to translate the
nonprocedural request of a set-based SQL statement into a procedure that can execute efficiently and return the
desired results. Unless I indicate otherwise, when I talk about compilation I will mean both the compilation of the
Transact-SQL and the optimization of the SQL statements.

After the plan is generated, it is placed in cache and then executed. I'll tell you more about the reuse of plans and what
happens during execution later in the chapter.

Compilation
During the compilation process, SQL Server parses each statement and creates something called a sequence tree.
This is one of the few data structures in SQL Server 2000 that has survived from SQL Server 6.5. The sequence tree
is then normalized. The main function of the normalizer is to perform binding, which includes verifying that the tables
and columns exist and loading the metadata for the tables and columns. Information about implicit conversions is also
added to the sequence tree; for example, if the query is trying to add 10.0 to an integer value, SQL Server will insert
an implicit convert into the tree. Normalization also replaces references to a view with the definition of that view.
This document is created with the unregistered version of CHM2PDF Pilot

Finally, normalization includes a few syntax-based optimizations. If the statement is a DML statement, SQL Server
extracts information from the sequence tree about that query and creates a special structure called a query graph,
which is set up to let the optimizer work on it effectively. The query graph is then optimized and a plan is produced.

Figure 15-1 shows the flow of this compilation process.

Figure 15-1. The steps in compiling a statement.

Optimization
SQL Server's query optimizer is a cost-based optimizer, which means that it tries to come up with the cheapest
execution plan for each SQL statement. Each possible execution plan has an associated cost in terms of the amount of
computing resources used. The query optimizer must analyze the possible plans and choose the one with the lowest
estimated cost. Some complex SELECT statements have thousands of possible execution plans. In these cases, the
query optimizer does not analyze all possible combinations. Instead, it tries to find an execution plan that has a cost
reasonably close to the theoretical minimum. Later in this section, I'll tell you about some of the ways that the
optimizer can reduce the amount of time it needs to spend on optimization.

The lowest estimated cost is not simply the lowest resource cost; the query optimizer chooses the plan that most
quickly returns results to the user with a reasonable cost in resources. For example, processing a query in parallel
(using multiple CPUs simultaneously for the same query) typically uses more resources than processing it serially using
a single CPU, but the query completes much faster. The optimizer will choose a parallel execution plan to return
results if the load on the server will not be adversely affected.

Optimization itself involves several steps. The first step is called trivial plan optimization. The idea behind trivial
plan optimization is that cost-based optimization is expensive to run. The optimizer can try many possible variations in
looking for the cheapest plan. If SQL Server knows that there is only one really viable plan for a query, it can avoid a
lot of work. A common example is a query that consists of an INSERT with a VALUES clause. There is only one
possible plan. Another example is a SELECT for which all the columns are among the keys of a unique composite
index and that index is the only one that is relevant. No other index has that set of columns in it. In these two cases,
SQL Server should just generate the plan and not try to find something better. The trivial plan optimizer finds the really
obvious plans that are typically very inexpensive. This saves the optimizer from having to consider every possible plan,
which can be costly and can outweigh any benefit provided by well-optimized queries.

If the trivial plan optimizer doesn't find a simple plan, SQL Server can perform some simplifications, which are usually
syntactic transformations of the query itself, to look for commutative properties and operations that can be
rearranged. SQL Server can perform operations that don't require looking at the cost or analyzing what indexes are
available but that can result in a more efficient query. SQL Server then loads up the metadata, including the statistical
information on the indexes, and then the optimizer goes through a series of phases of cost-based optimization.
This document is created with the unregistered version of CHM2PDF Pilot

The cost-based optimizer is designed as a set of transformation rules that try various permutations of indexes and join
strategies. Because of the number of potential plans in SQL Server 2000, if the optimizer just ran through all the
combinations and produced a plan, the optimization process would take a long time. So optimization is broken up into
phases. Each phase is a set of rules. After each phase, SQL Server evaluates the cost of any resulting plan, and if the
plan is cheap enough, it executes that plan. If the plan is not cheap enough, the optimizer runs the next phase, which is
another set of rules. In the vast majority of cases, SQL Server finds a good plan in the preliminary phases.

If the optimizer goes through all the preliminary phases and still hasn't found a cheap plan, it examines the cost of the
best plan it has so far. If the cost is above a particular threshold, the optimizer goes into a phase called full
optimization. You can configure this threshold using the cost threshold for parallelism configuration option. The full
optimization phase assumes that this plan should be run in parallel. However, the optimizer will not consider this option
at all if the machine only has a single CPU available for SQL Server's use. If the machine is very busy, SQL Server
might decide at execution time to run the plan on a single CPU, but the optimizer tries to produce a good parallel plan.
If the cost is below the parallelism threshold, the full optimization phase just uses a brute force method to find a serial
plan, checking every possible combination of indexes and processing strategies. Figure 15-2 shows the flow of
processing through the optimizer.

Figure 15-2. The steps in optimizing a statement.

How The Query Optimizer Works

Monitoring and tuning of queries are essential steps in optimizing performance. Knowing how the query optimizer
works can be helpful as you think about how to write a good query or what indexes to define. However, you should
guard against outsmarting yourself and trying to predict what the optimizer will do. You might miss some good options
this way. Try to write your queries in the most intuitive way you can and then try to tune them only if their performance
doesn't seem good enough.

Nevertheless, insight into how the optimizer works is certainly useful. It can help you understand why certain indexes
are recommended over others and it can help you understand the query plan output in SQL Query Analyzer. For
each table involved in the query, the query optimizer evaluates the search conditions in the WHERE clause and
considers which indexes are available to narrow the scan of a table. That is, the optimizer evaluates to what extent an
index can exclude rows from consideration. The more rows that can be excluded, the better, because that leaves
fewer rows to process.

Joins can be processed using one of three methods: nested iteration, hashing, or merging. For any of these methods,
the optimizer decides on the order in which the tables should be accessed. Because a nested iteration is a loop, order
This document is created with the unregistered version of CHM2PDF Pilot

is important. The fewer the iterations through the loops, the less processing will be required. So it is useful to start with
the table (or tables) that can exclude the most rows as the outer loops. The general strategy is to have the outer table
limit the search the most, which results in the fewest total iterations (scans). With hash or merge joins, SQL Server
builds an in-memory structure from one or both of the tables. To conserve memory resources, it tries to determine
which is the smaller table or which will have the smallest number of qualifying rows after the WHERE conditions are
applied. This table is typically chosen as the first table to be processed. We'll look at each of these join strategies
again later in the chapter.

For each combination of join strategy, join order, and indexes, the query optimizer estimates a cost, taking into
account the number of logical reads and the memory resources that are required and available. Then the optimizer
compares the cost of each plan and chooses the plan with the lowest estimate. You can think of query optimization as
happening in three main steps: query analysis, index selection, and join selection, although there is a lot of overlap
between the index selection and join selection steps. The following sections discuss each step.

Query Analysis
During the query analysis phase, the query optimizer looks at each clause of the query and determines whether it can
be useful in limiting how much data must be scanned—that is, whether the clause is useful as a search argument
(SARG) or as part of the join criteria. A clause that can be used as a search argument is referred to as sargable, or
optimizable, and can make use of an index for faster retrieval.

A SARG limits a search because it specifies an exact match, a range of values, or a conjunction of two or more items
joined by AND. A SARG contains a constant expression (or a variable that is resolved to a constant) that acts on a
column by using an operator. It has the form
column inclusive_operator <constant or variable>

or
<constant or variable> inclusive_operator column

The column name can appear on one side of the operator, and the constant or variable can appear on the other side.
If a column appears on both sides of the operator, the clause is not sargable. Sargable operators include =, >, <, =>,
<=, BETWEEN, and sometimes LIKE. Whether LIKE is sargable depends on the type of wildcards (regular
expressions) used. For example, LIKE 'Jon% is sargable but LIKE '?Jon' is not because the wildcard (%) at the
beginning prevents the use of an index. Here are some SARG examples:
name = 'jones'

salary > 40000

60000 < salary

department = 'sales'

name = 'jones' AND salary > 100000

name LIKE 'dail%'

A single SARG can include many conditions if they are AND'ed together. That is, one index might be able to operate
on all the conditions that are AND'ed together. In the example above, there might be an index on (name, salary), so
the entire clause name = jones AND salary > 100000 can be considered one SARG and can be evaluated for
qualifying rows using one index. If OR is used instead of AND in this example, a single index scan cannot be used to
qualify both terms. The reason should be clear—if the lead field in the index key is name, the index is useful for finding
This document is created with the unregistered version of CHM2PDF Pilot

just the 'jones' rows. If the criterion in the WHERE clause includes AND salary > 100000, the second field of the
index also qualifies those rows. That is, all the rows that have a name value of 'jones' can be further restricted to find
the rows with 'jones' and an appropriate salary. The set of rows that meets both of the AND conditions is a subset of
the rows that meet just the first condition on name. But if the second criterion is OR salary > 100000, the index is
not useful in the same way because all the rows would need to be examined, not just the 'jones' entries. The set of
rows that meet either one of the conditions is a superset of the rows that meet just the first condition on name.

A phone book analogy can also be useful. If you want to find people who have the name "jones" AND live on 5th
Avenue, using the phone book can help greatly reduce the size of your search. But if you want to find people who
have the name "Jones" OR live on 5th Avenue, you have to scan every entry in the entire phone book. This assumes
that you have only one phone book that is sorted alphabetically by name. If you have two phone books, one sorted
by name and one sorted by street, that's another story, which I'll discuss a bit later.

An expression that is not sargable cannot limit the search. That is, SQL Server must evaluate every row to determine
whether it meets the conditions in the WHERE clause. So an index is not useful to nonsargable expressions. Typical
nonsargable expressions include negation operators such as NOT, !=, <>, !>, !<, NOT EXISTS, NOT IN, and
NOT LIKE. Don't extrapolate too far and think that this means using a nonsargable clause always results in a table
scan. An index is not useful to the nonsargable clause, but there might be indexes useful to other SARGs in the query.
Queries often have multiple clauses, so a great index for one or more of the other clauses might be available. Here are
some examples of nonsargable clauses:
ABS(price) < 4

name LIKE '%jon%'

name = 'jones' OR salary > 100000

NOTE
The last example above is not a single search argument, but each expression on either side of the OR is individually
sargable. So a single index won't be used to evaluate both expressions, as it might if the operator is AND. A separate
index can still be useful to each expression.

Expressions involving computations might be sargable because SQL Server 2000 can do scalar simplification in some
cases. The following table shows examples of expressions that the query optimizer will simplify, along with the
resultant simplified expression that is used to determine the usefulness of an index.
Original Expression Simplified Expression
WHERE price * 12 = costs WHERE price = costs/12
WHERE salary * 1 > 40000 WHERE salary > 40000

The simplified expression is not guaranteed to be exactly equivalent to the original, particularly if you have a mixture of
datatypes and implicit conversions are occurring. However, the simplified expression is used only during optimization,
to detect and measure the usefulness of a particular index. When determining whether a particular row actually
qualifies for the result set, SQL Server always uses the original expression.

Let's look at a specific example. The Northwind database includes a table called Order Details, which has a
nonclustered index on the column ProductID. When a query contains a SARG involving ProductID, the query
optimizer has to determine whether using the nonclustered index will be the fastest way to access the qualifying rows.
Remember that a nonclustered index contains every key value in its leaf level, with a bookmark pointing to the actual
location of the row having that key. If too many rows satisfy the SARG's condition, the optimizer might decide it's too
expensive to use the index and opt instead to scan the entire table. In addition to information about the key columns
and the type of each index, the sysindexes system table contains a column called statblob, which holds statistical
information about the distribution of all the different key values. We'll look at statistics in detail a bit later, but for now,
be aware that examining the statistics is part of determining whether an index is useful.
This document is created with the unregistered version of CHM2PDF Pilot

The first query below has a SARG involving the ProductID column, and SQL Server is able to use the statistical
information to estimate that five rows will meet the condition in the WHERE clause. The optimizer will choose to use
the index on ProductID.
USE Northwind
SELECT * FROM [Order Details]
WHERE ProductID = 9

In the next query, because of the presence of the function ABS, there is no SARG and the index on ProductID is not
even considered. The optimizer has no real choice other than to scan the entire table.
USE Northwind
SELECT * FROM [Order Details]
WHERE ABS(ProductID) = 9

In the third example, the optimizer can internally convert the condition in the WHERE clause into a SARG and
correctly estimate the number of rows. However, the conversion is used only during optimization. During execution,
the conversion cannot be done while traversing the index. The optimizer knows that SQL Server won't be able to do
the repeated multiplication by 2 at every step in the index tree, so the final plan is to do a table scan. However, the
optimizer used the index to come up with an estimate of the number of rows. In more complex queries, even if the
index itself can't be used to locate the rows, knowing the number of qualifying rows can be useful in making other
decisions about the best plan. The crucial point here is that when you used the function, the optimizer totally ignored
the index. When you used an expression that was convertible to a SARG, the optimizer could examine the statistics,
even if the index itself couldn't be used during execution.
USE Northwind
SELECT * FROM [Order Details]
WHERE ProductID * 2 = 18

How can you know what the plan is or what the optimizer is estimating for the number of rows? SQL Server 2000
provides several tools to help you analyze the optimizer's choices, and I'll go into much more detail on these tools in
Chapter 16 when I tell you about tuning your queries. For now, let's just have a sneak preview of how to look at a
graphical representation of the plan that the optimizer has chosen. If you type any of the three queries above into the
SQL Query Analyzer query window, you can highlight the query and click the Display Estimated Execution Plan
button just to right of the database list. Alternatively, you can go to the Query menu and choose Display Estimated
Execution Plan. Figure 15-3 shows the plan for the first query. If you place your cursor over the rightmost icon, you'll
see the yellow box of details appear. The important thing to note is that the operation is Index Seek. Index Seek
means that SQL Server will traverse the index from the root down to the leaf level, comparing the values in the SARG
to the key values in the index rows to determine which page to look at next. Seeking through an index typically means
a root-to-leaf traversal. Note also in Figure 15-3 that the estimated row count is five.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 15-3. A graphical query plan for an index seek operation.

Figure 15-4 shows the plan for the second query, and you can see that the operation is Clustered Index Scan. A
scan, the alternative to Index Seek, means that SQL Server will stay at the leaf level and traverse just that one level of
the index from the first page to the last. I like to think of a seek as a vertical traversal of an index and a scan as a
horizontal traversal. Remember that for a clustered index, the leaf level of the index is the table data itself, so scanning
the clustered index really means scanning the table. In fact, the only time you'll see Table Scan as an operator in the
plan is for heaps. Also notice in Figure 15-4 that the row count is nowhere close to reality. The index statistics were
not used, and the optimizer just had to take a wild guess. I'll tell you a bit more about how and when the optimizer can
"guess" later in this chapter.

Figure 15-4. A graphical query plan for a clustered index scan (table scan).

Finally, Figure 15-5 shows the plan for the third query. Note that the row count is the same as in the first query,
which means that the statistics can be used. But the operation to be performed is the index scan of the clustered index.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 15-5. A graphical query plan for a nonclustered index scan.

As you saw in the second example, expressions that apply a function to a column are not sargable, and in SQL
Server 2000, the optimizer does not attempt to internally convert them to something that is sargable. In some cases,
however, you might be able to write an equivalent expression. For example, the Orders table has an index on the
CustomerID column, which is a character string. The following two queries return the same three rows in their result
set, but the first one does not use an index and the second one does:
SELECT * FROM Orders WHERE SUBSTRING(CustomerID, 1, 1) = 'N'

SELECT * FROM Orders WHERE CustomerID LIKE 'N%'

Future versions of SQL Server will address more issues of nonsargable clauses.

Index Selection
During the second phase of query optimization, index selection, the query optimizer determines whether an index
exists for a sargable clause, assesses the index's usefulness by determining the selectivity of the clause (that is, how
many rows will be returned), and estimates the cost of finding the qualifying rows. An index is potentially useful if its
first column is used in the search argument and the search argument establishes a lower bound, upper bound, or both
to limit the search. In addition, if an index contains every column referenced in a query, even if none of those columns
is the first column of the index, the index is considered useful.

An index that contains all the referenced columns is called a covering index and is one of the fastest ways to access
data. The data pages do not have to be accessed at all, which can mean a substantial savings in physical I/O. For
example, suppose that the Employees table has an index on last name, first name, and date of hire. The following
query is covered by this index:
SELECT LastName, HireDate
FROM Employees
WHERE FirstName = 'Sven'

You might have more covering indexes than you're aware of. In SQL Server 2000, a nonclustered index contains the
clustered index key as part of the bookmark. So if the Employees table has a clustered index on the employee ID (
EmployeeID) column, the nonclustered index described earlier, on LastName, FirstName, and HireDate, will also
contain the EmployeeID for every data row, stored in the leaf level of the nonclustered index. Thus, the following is
also a covered query:
This document is created with the unregistered version of CHM2PDF Pilot

SELECT EmployeeID, FirstName, LastName

FROM Employees
WHERE LastName LIKE 'B%'

Index Statistics
After the query optimizer finds a potentially useful index that matches the clause, it evaluates the index based on the
selectivity of the clause. The optimizer checks the index's statistics—the histogram of values accessible through the
index's row in the sysindexes table. The histogram is created when the index is created on existing data, and the
values are refreshed each time UPDATE STATISTICS runs. If the index is created before data exists in the table, no
statistics appear. The statistics will be misleading if they were generated when the dispersion of data values was
significantly different from what appears in the current data in the table. However, SQL Server detects whether
statistics are not up-to-date, and by default it automatically updates them during query optimization.

Statistics are a histogram consisting of an even sampling of values for the index key (or the first column of the key for
a composite index) based on the current data. The histogram is stored in the statblob field of the sysindexes table,
which is of type image. (As you know, image data is actually stored in structures separate from the data row itself.
The data row merely contains a pointer to the image data. For simplicity's sake, I'll talk about the index statistics as
being stored in the image field called statblob.) To fully estimate the usefulness of an index, the optimizer also needs
to know the number of pages in the table or index; this information is stored in the dpages column of sysindexes.

The statistical information also includes details about the uniqueness (or density) of the data values encountered,
which provides a measure of how selective the index is. Recall that the more selective an index is, the more useful it is,
because higher selectivity means that more rows can be eliminated from consideration. A unique index, of course, is
the most selective—by definition, each index entry can point to only one row. A unique index has a density value of 1/
number of rows in the table.

Density values range from 0 through 1. Highly selective indexes have density values of 0.10 or lower. For example, a
unique index on a table with 8345 rows has a density of 0.00012 (1/8345). If a nonunique nonclustered index has a
density of 0.2165 on the same table, each index key can be expected to point to about 1807 rows (0.2165 × 8345).
This is probably not selective enough to be more efficient than just scanning the table, so this index is probably not
useful. Because driving the query from a nonclustered index means that the pages must be retrieved in index order, an
estimated 1807 data page accesses (or logical reads) are needed if there is no clustered index on the table and the
leaf level of the index contains the actual RID of the desired data row. The only time a data page doesn't need to be
reaccessed is when the occasional coincidence occurs in which two adjacent index entries happen to point to the
same data page.

Assume in this example that about 40 rows fit on a page, so there are about 209 total pages. The chance of two
adjacent index entries pointing to the same page is only about 0.5 percent (1/209). The number of logical reads for
just the data pages, not even counting the index I/O, is likely to be close to 1807. If there is also a clustered index on
the table, a couple of additional page accesses will be needed for each nonclustered key accessed. In contrast, the
entire table can be scanned with just 209 logical reads—the number of pages in the table. In this case, the optimizer
looks for better indexes or decides that a table scan is the best it can do.

The statistics histogram records steps (samples) for only the lead column of the index. This optimization takes into
account that an index is useful only if its lead column is specified in a WHERE clause. However, density information is
stored for multiple combinations of columns in a composite index. For example, if we have an index on (lastname,
firstname), the statblob field will contain a density value for lastname and another density value for the combination
of lastname and firstname. This will become clearer when we look at some actual statistics information.

SQL Server defines density as follows:

density = 1/cardinality of index keys
This document is created with the unregistered version of CHM2PDF Pilot

The cardinality means how many unique values exist in the data. Statistics for a single column index consist of one
histogram and one density value. The multicolumn statistics for one set of columns in a composite index consist of one
histogram for the first column in the index and density values for each prefix combination of columns (including the
first column alone). The fact that density information is kept for all columns helps the optimizer decide how useful the
index is for joins.

Suppose, for example, that an index is composed of three key fields. The density on the first column might be 0.50,
which is not too useful. But as you look at more key columns in the index, the number of rows pointed to is fewer than
(or in the worst case, the same as) the first column, so the density value goes down. If you're looking at both the first
and second columns, the density might be 0.25, which is somewhat better. And if you examine three columns, the
density might be 0.03, which is highly selective. It doesn't make sense to refer to the density of only the second
column. The lead column density is always needed.

The histogram contains up to 200 values of a given key column. In addition to the histogram, the statblob field
contains the following information:
 The time of the last statistics collection
 The number of rows used to produce the histogram and density information
 The average key length
 Densities for other combinations of columns

The following example uses the Orders table in the Northwind database. Assume that we have built a nonclustered
composite index (cust_date_indx) on CustomerID and OrderDate.
CREATE INDEX cust_date_indx ON Orders(CustomerID, OrderDate)

The existing clustered index is on OrderID, so OrderID is considered part of the key in every nonclustered index
row. We can use DBCC SHOW_STATISTICS to display the statistics information for this index:
DBCC SHOW_STATISTICS('Orders', 'cust_date_indx')

Here are the results:

Statistics for INDEX 'cust_date_indx'.
Updated Rows Rows Sampled Steps Density Average key length
----------- ------ ------------- ------ -------- ------------------
Aug 31 2000 830 830 89 0.0 22.0

All density Average Length Columns

--------------- --------------- ------------------------------
1.1235955E-2 10.0 CustomerID
1.2150669E-3 18.0 CustomerID, OrderDate
1.2048193E-3 22.0 CustomerID, OrderDate, OrderID

RANGE_HI_KEY RANGE_ROWS EQ_ROWS DISTINCT_RANGE_ROWS AVG_RANGE_ROWS

------------ ----------- ------- -------------------- --------------
ALFKI 0.0 6.0 0 0.0
ANATR 0.0 4.0 0 0.0
ANTON 0.0 7.0 0 0.0
AROUT 0.0 13.0 0 0.0
BERGS 0.0 18.0 0 0.0
BLAUS 0.0 7.0 0 0.0
BLONP 0.0 11.0 0 0.0
BOLID 0.0 3.0 0 0.0
BONAP 0.0 17.0 0 0.0
BOTTM 0.0 14.0 0 0.0
BSBEV 0.0 10.0 0 0.0
CACTU 0.0 6.0 0 0.0
This document is created with the unregistered version of CHM2PDF Pilot

CENTC 0.0 1.0 0 0.0

CHOPS 0.0 8.0 0 0.0
COMMI 0.0 5.0 0 0.0
CONSH 0.0 3.0 0 0.0
DRACD 0.0 6.0 0 0.0
DUMON 0.0 4.0 0 0.0
EASTC 0.0 8.0 0 0.0
ERNSH 0.0 30.0 0 0.0
FAMIA 0.0 7.0 0 0.0
FOLIG 0.0 5.0 0 0.0
FOLKO 0.0 19.0 0 0.0
FRANK 0.0 15.0 0 0.0
FRANR 0.0 3.0 0 0.0
FRANS 0.0 6.0 0 0.0
FURIB 0.0 8.0 0 0.0
GALED 0.0 5.0 0 0.0
GODOS 0.0 10.0 0 0.0
GOURL 0.0 9.0 0 0.0
GREAL 0.0 11.0 0 0.0
GROSR 0.0 2.0 0 0.0
HANAR 0.0 14.0 0 0.0
HILAA 0.0 18.0 0 0.0
HUNGC 0.0 5.0 0 0.0
HUNGO 0.0 19.0 0 0.0
ISLAT 0.0 10.0 0 0.0
KOENE 0.0 14.0 0 0.0
LACOR 0.0 4.0 0 0.0
LAMAI 0.0 14.0 0 0.0
LAUGB 0.0 3.0 0 0.0
LAZYK 0.0 2.0 0 0.0
LEHMS 0.0 15.0 0 0.0
LETSS 0.0 4.0 0 0.0
LILAS 0.0 14.0 0 0.0
LINOD 0.0 12.0 0 0.0
LONEP 0.0 8.0 0 0.0
MAGAA 0.0 10.0 0 0.0
MAISD 0.0 7.0 0 0.0
MEREP 0.0 13.0 0 0.0
MORGK 0.0 5.0 0 0.0
NORTS 0.0 3.0 0 0.0
OCEAN 0.0 5.0 0 0.0
OLDWO 0.0 10.0 0 0.0
OTTIK 0.0 10.0 0 0.0
PERIC 0.0 6.0 0 0.0
PICCO 0.0 10.0 0 0.0
PRINI 0.0 5.0 0 0.0
QUEDE 0.0 9.0 0 0.0
QUEEN 0.0 13.0 0 0.0
QUICK 0.0 28.0 0 0.0
RANCH 0.0 5.0 0 0.0
RATTC 0.0 18.0 0 0.0
REGGC 0.0 12.0 0 0.0
RICAR 0.0 11.0 0 0.0
RICSU 0.0 10.0 0 0.0
ROMEY 0.0 5.0 0 0.0
SANTG 0.0 6.0 0 0.0
SAVEA 0.0 31.0 0 0.0
SEVES 0.0 9.0 0 0.0
SIMOB 0.0 7.0 0 0.0
SPECD 0.0 4.0 0 0.0
SPLIR 0.0 9.0 0 0.0
SUPRD 0.0 12.0 0 0.0
THEBI 0.0 4.0 0 0.0
THECR 0.0 3.0 0 0.0
TOMSP 0.0 6.0 0 0.0
This document is created with the unregistered version of CHM2PDF Pilot

TORTU 0.0 10.0 0 0.0

TRADH 0.0 6.0 0 0.0
TRAIH 0.0 3.0 0 0.0
VAFFE 0.0 11.0 0 0.0
VICTE 0.0 10.0 0 0.0
VINET 0.0 5.0 0 0.0
WANDK 0.0 10.0 0 0.0
WARTH 0.0 15.0 0 0.0
WELLI 0.0 9.0 0 0.0
WHITC 0.0 14.0 0 0.0
WILMK 0.0 7.0 0 0.0
WOLZA 0.0 7.0 0 0.0

This output indicates that the statistics for this index were last updated on August 31, 2000. It also indicates that the
table currently has 830 rows and that 89 different samples were used. The average key length in the first row is for all
columns of the index.

The next section shows the density values for each left-based subset of key columns. There are 89 different values
for CustomerID, so the density is 1/89 or 1.1235955E-2 or 0.011235955. There are 823 different values for the
combination of CustomerID and OrderDate, so the density is 1/823 or 1.2150669E-3 or 0.0012150669. When the
third key column of OrderID is included, the key is now unique, so the density is 1 divided by the number of rows in
the name, or 1/830, which is 1.2048193E-3 or 0.0012048193.

In the statblob column, up to 200 sample values are stored; the range of key values between each sample value is
called a step. The sample value is the endpoint of the range. Three values are stored along with each step: a value
called EQ_ROWS, which is the number of rows that have a value equal to that sample value; a value called
RANGE_ROWS, which specifies how many other values are inside the range (between two adjacent sample values);
and the number of distinct values, or RANGE_DENSITY of the range. The DBCC SHOW_STATISTICS output
shows us the first two of these three values, but not the range density. The RANGE_DENSITY is instead used to
compute two additional values:
 DISTINCT_RANGE_ROWS—the number of distinct rows inside this range (not counting the
RANGE_HI_KEY value itself). This is computed as 1/RANGE_DENSITY.
 AVG_RANGE_ROWS—the average number of rows per distinct value, computed as RANGE_DENSITY
* RANGE_ROWS.

In the output above, all the distinct key values in the first column of the index were one of the sample values, so there
were no additional values in between the range endpoints. That's why so many of the values are 0 in the output.

In addition to statistics on indexes, SQL Server can also keep track of statistics on columns with no indexes.
Knowing the density, or the likelihood of a particular value occurring, can help the optimizer determine an optimum
processing strategy, even if SQL Server can't use an index to actually locate the values. I'll discuss column statistics in
more detail when we look at statistics management later in this section.

Query optimization is probability based, which means that its conclusions can be wrong. The optimizer can make
decisions that are not optimal even if they make sense from a probability standpoint. (As an analogy, you can think of
national election polls, which are based on relatively small samples. The famous headline "Dewey Defeats Truman!"
proves that such statistical sampling can be wrong.)
NOTE
The above information on SHOW_STATISTICS values was adapted from a preliminary copy of a whitepaper by
Lubor Kollar. I'm indebted to him for his assistance.

Index Cost
The second part of determining the selectivity of a SARG is calculating the estimated cost of the access methods that
can be used. Even if a useful index is present, it might not be used if the optimizer determines that it is not the cheapest
This document is created with the unregistered version of CHM2PDF Pilot

access method. One of the main components of the cost calculation, especially for queries involving only a single
table, is the amount of logical I/O, which is the number of page accesses that are needed. Logical I/O is counted
whether the pages are already in the memory cache or must be read from disk and brought into the cache. The term
logical I/O is sometimes used to refer to I/O from cache, as if a read were either logical or physical, but this is a
misnomer. As discussed in Chapter 3, pages are always retrieved from the cache via a request to the buffer manager,
so all reads are logical. If the pages are not already in the cache, they must be brought in first by the buffer manager.
In those cases, the read is also physical. Only the buffer manager, which serves up the pages, knows whether a page
is already in cache or must be accessed from disk and brought into cache. The ratio of the I/O already in the cache to
the logical I/O is referred to as the cache-hit ratio, an important metric to watch.

The optimizer evaluates indexes to estimate the number of likely "hits" based on the density and step values in the
statistics. Based on these values, it estimates how many rows qualify for the given SARG and how many logical reads
would retrieve those qualifying rows. It might find multiple indexes that can find qualifying rows, or it might determine
that just scanning the table and checking all the rows is best. It tries to find the access method that will require the
fewest logical reads.

Whether the access method uses a clustered index, one or more nonclustered indexes, a table scan, or another option
determines the estimated number of logical reads. This number can be very different for each method. Using an index
to find qualifying rows is frequently referred to as an index driving the scan. When a nonclustered index drives the
scan, the query optimizer assumes that the page containing each identified qualifying row is probably not the same
page accessed the last time. Because the pages must be retrieved according to the order of the index entries,
retrieving the next row with a query driven by a nonclustered index will likely require that a different page than the one
that contained the previous row be fetched because the two rows probably don't reside on the same page. They might
reside on the same page by coincidence, but the optimizer correctly assumes that this will typically not be the case.
(With 5000 pages, the chance of this occurring is 1/5000.) If the index is not clustered, data order is random with
respect to the index key.

If the scan is driven from a clustered index, the next row is probably located on the same page as the previous row
because the index leaf is, in fact, the data. The only time this is not the case when you use a clustered index is when
the last row on a page has been retrieved; the next page must be accessed, and it will contain the next bunch of rows.
But moving back and forth between the index and the data pages is not required.

There is, of course, a chance that a specific page might already be in cache when you use a nonclustered index. But a
logical read will still occur even if the scan does not require physical I/O (a read from disk). Physical I/O is an order
of magnitude more costly than I/O from cache. But don't assume that only the cost of physical I/O matters—reads
from the cache are still far from free. To minimize logical I/O, the query optimizer tries to produce a plan that will
result in the fewest number of page operations. In doing so, the optimizer will probably minimize physical I/O as well.
For example, suppose we need to resolve a query with the following SARG:
WHERE Emp_Name BETWEEN 'Smith' AND 'Snow'

Here are the relevant index entries for a nonclustered index on Emp_Name:
Index_Key (Emp_Name) Bookmarks: Found on Page(s)
-------------------- -------------------
Smith 1:267, 1:384, 1:512
Smyth 1:267
Snow 1:384, 1:512

If this table is a heap, when we use this nonclustered index, six logical reads are required to retrieve the data pages in
addition to the I/O necessary to read the index. If none of the pages is already cached, the data access results in three
physical I/O reads, assuming that the pages remain in the cache long enough to be reused for the subsequent rows
(which is a good bet).
This document is created with the unregistered version of CHM2PDF Pilot

Suppose that all the index entries are on one leaf page and the index has one intermediate level. In this case, three I/O
reads (logical and probably physical) are necessary to read the index (one root level, one intermediate level, and one
leaf level). So chances are that driving this query via the nonclustered index requires about nine logical I/O reads, six
of which are probably physical if the cache started out empty. The data pages are retrieved in the following order:
Page 1:267 to get row with Smith
Page 1:384 for Smith
Page 1:512 for Smith
Page 1:267 (again) for Smyth
Page 1:384 (again) for Snow
Page 1:512 (again) for Snow

The number of logical reads would be more if the table also had a clustered index (for example, on the zip code
field). Our leaf-level index entries might look like this:
Index_Key (Emp_Name) Bookmarks: Clustered Key(s)
-------------------- ---------------------------
Smith 06403, 20191, 98370
Smyth 37027
Snow 22241, 80863

For each of the six nonclustered keys, we have to traverse the clustered index. You typically see about three to four
times as many logical reads for traversing a nonclustered index for a table that also has a clustered index. However, if
the indexes for this table are re-created so that a clustered index exists on the Emp_Name column, all six of the
qualifying rows are probably located on the same page. The number of logical reads is probably only three (the index
root, the intermediate index page, and the leaf index page, which is the data page), plus the final read that will retrieve
all the qualifying rows. This scenario should make it clear to you why a clustered index can be so important to a range
query.

With no clustered index, the optimizer has to choose between a plan that does a table scan and a plan that uses one
or more nonclustered indexes. The number of logical I/O operations required for a scan of the table is equal to the
number of pages in the table. A table scan starts at the first page and uses the Index Allocation Map (IAM) to access
all the pages in the table. As the pages are read, the rows are evaluated to see whether they qualify based on the
search criteria. Clearly, if the table from the previous example had fewer than nine total data pages, fewer logical
reads would be needed to scan the whole table than to drive the scan off the nonclustered index, which was estimated
to take nine logical reads.

The estimated logical reads for scanning qualifying rows is summarized in Table 15-1. The access method with the
least estimated cost is chosen based on this information.

Table 15-1. The cost of data access for tables with different index structures.
Access Method Estimated Cost (in Logical Reads)*
Table scan The total number of data pages in the table.
The number of levels in the index plus the number of data
Clustered index pages to scan. (Data pages to be scanned = number of
qualifying rows / rows per data page.)
The number of levels in the index plus the number of leaf
pages plus the number of qualifying rows (a logical read
for the data page of each qualifying row). The same data
Nonclustered index on a heap
pages are often retrieved (from cache) many times, so the
number of logical reads can be much higher than the
number of pages in the table.
This document is created with the unregistered version of CHM2PDF Pilot

The number of levels in the index plus the with a clustered

Nonclustered index on a table index number of leaf pages plus the number of qualifying
rows times the cost of searching for a clustered index key.
The number of levels in the index plus the number of leaf
index pages (qualifying rows / rows per leaf page). The
Covering nonclustered index
data page need not be accessed because all necessary
information is in the index key.

* All computations are rounded up to the next integer

Using Multiple Indexes

The query optimizer can decide to use two or more nonclustered indexes to satisfy a single query. When more than
one index is considered useful because of two or more SARGs, the cost estimate changes a bit from the formulas
given in Table 15-1. In addition to the I/O cost of finding the qualifying rows in each index, there is an additional cost
of finding the intersection of the indexes so that SQL Server can determine which data rows satisfy all search
conditions. Since each nonclustered index key contains a locator for the actual data, the results of the two index
searches can be treated as worktables and joined together on the locator information.

Suppose we have the following query and we have indexes on both the LastName and FirstName columns. In this
case, we have a clustered index on ID number, which is a unique value.
SELECT FirstName, LastName, EmployeeID FROM Employees
WHERE LastName BETWEEN 'Smith' AND 'Snow'
AND FirstName BETWEEN 'Daniel' and 'David'

If the query optimizer decides to use both indexes, SQL Server will build two worktables with the results of each
index search:
Index_Key (LastName) Bookmark(Clustered Key)
-------------------- ----------------------
Smith 66-712403
Smith 87-120191
Smith 76-137027
Smyth 11-983770
Snow 43-220191
Snow 21-780863

Index_Key (FirstName) Bookmark(Clustered Key)

-------------------- ----------------------
Daniel 11-983770
Daniel 77-321456
Danielle 66-712403
David 29-331218

These worktables are joined on the row locator information, which is unique for each row. Remember, even if you
don't declare a clustered index as unique, SQL Server adds a special uniqueifier where needed so there is always a
unique row locator in every nonclustered index. From the information above, we can see that only two rows have
both their first name and last name values in the requested range. The result set is:
FirstName LastName EmployeeID
---------- --------------- -----------
Daniel Smythe 11-983770
Danielle Smith 66-712403
This document is created with the unregistered version of CHM2PDF Pilot

In this example, the data in the two indexes is enough to satisfy the query. If we also want to see information such as
address and phone number, the query optimizer uses the bookmark to find the actual data row by traversing the
clustered index. It can also use multiple indexes if a single table has multiple SARGs that are OR'ed together. In this
case, SQL Server finds the UNION of the rows returned by each index to determine all the qualifying rows. You can
see the main differences if you take the same query and data and write an OR query instead of an AND:
SELECT FirstName, LastName, EmployeeID FROM Employees
WHERE LastName BETWEEN 'Smith' AND 'Snow'
OR FirstName BETWEEN 'Daniel' AND 'David'

SQL Server can use the two indexes in the same way and build the same two worktables. However, it must UNION
the two worktables together and remove duplicates. Once it removes the duplicates, it uses the bookmarks to find the
rows in the table. All the result rows will have either a first name or a last name in the requested range, but not
necessarily both. For example, we can return a row for Daniel Delaney or Bridgette Smith. If we don't remove
duplicates, we end up with Daniel Smythe and Danielle Smith showing up twice because they meet both conditions of
the SARGs.

Unusable Statistics
In two cases, the query optimizer can't use the statistics to estimate how many rows will satisfy a given SARG. First,
if there are no statistics available, SQL Server obviously can't come up with any cost estimate. However, this situation
is rare in SQL Server 2000 because by default every database has the two database options auto create statistics
and auto update statistics set to TRUE. As you'll see later, you can turn this behavior off at the table level or the
database level, but this is usually not recommended. The second situation in which there are no usable statistics is
when the values in a SARG are variables. Consider this example:
DECLARE @name varchar(30)
SET @name = 'Zelda'
SELECT FirstName, LastName, EmployeeID FROM Employees
WHERE LastName > @name

When the preceding query is optimized, the SET statement has not been executed and the value of @name is
unknown. No specific value can be used to compare the steps in the index's statistics information, so the query
optimizer must guess.
NOTE
Don't confuse variables with parameters, even though their syntax is nearly identical. A variable's value is never known
until the statement is actually executed; it is never known at compile time. Stored procedure parameters are known
when the procedure is compiled because compilation and optimization don't even take place until the procedure is
actually called with specific values for the parameters.

If the query optimizer can't use the statistics for either of the reasons mentioned, the server uses fixed percentages,
shown here, depending on the operator. These fixed percentages can be grossly inaccurate for your specific data,
however, so make sure that your statistics are up-to-date.
Operator Percentage of Rows
= 10
> 30
< 30
BETWEEN 10

When the SARG involves an equality, the situation is usually handled in a special way. The 10 percent estimate shown
above is actually used only in the unusual case in which there are no statistics at all. If there are statistics, they can be
used. Even if we don't have a particular value to compare against the steps in the statistics histogram, the density
This document is created with the unregistered version of CHM2PDF Pilot

information can be used. That information basically tells the query optimizer the expected number of duplicates, so
when the SARG involves looking for one specific value, it assumes that the value occurs the average number of times.

An even more special case occurs when the query optimizer recognizes an equality in the WHERE clause and the
index is unique. Because this combination yields an exact match and always returns at most one row, the query
optimizer doesn't have to use statistics. For queries of one row that use an exact match such as a lookup by primary
key, a unique nonclustered index is highly efficient. In fact, many environments probably shouldn't "waste" their
clustered index on the primary key. If access via the complete primary key is common, it might be beneficial to specify
NONCLUSTERED when you declare the primary key and save the clustered index for another type of access (such
as a range query) that can benefit more from the clustered index.

Join Selection
Join selection is the third major step in query optimization. If the query is a multiple-table query or a self-join, the
query optimizer evaluates join selection and selects the join strategy with the lowest cost. It determines the cost using
a number of factors, including the expected number of reads and the amount of memory required. It can choose
between three basic strategies for processing joins: nested loop joins, merge joins, and hash joins. In each method, the
tables to be joined (or subsets of tables that have already been restricted with conditions in the WHERE clause) are
called the join inputs.

Nested Loop Joins

Joins are frequently processed as nested iterations—a set of loops that take a row from the first table and use that
row to scan the inner table, and so on, until the result that matches is used to scan the last table. The number of
iterations through any of the loops equals the number of scans that must be done. This is not necessarily a table scan,
since it is frequently done using an index. In fact, if no useful index is available to be used for an inner table, nested
iteration probably isn't used and a hash join strategy is used instead. The result set is narrowed down as it progresses
from table to table within each iteration in the loop. If you've done programming with an ISAM-type product, this
should look familiar in terms of opening one "file" and then seeking into another in a loop. To process the following
join
SELECT <some columns>
FROM departments dept JOIN employees empl
ON dept.deptno=empl.deptno

you use pseudocode that looks something like this:

DO (until no more dept rows);
GET NEXT dept row;
{
begin
// Scan empl, hopefully using an index on empl.deptno
GET NEXT empl row for given dept
end
}

Merge Joins
Merge joins were introduced in SQL Server 7. You can use a merge join when the two join inputs are both sorted
on the join column. Consider the query we looked at in the previous section:
SELECT <some columns>
FROM departments dept JOIN employees empl
ON dept.deptno=empl.deptno
This document is created with the unregistered version of CHM2PDF Pilot

If both the departments table and the employees table have clustered indexes on the deptno column, the query is a
prime candidate for a merge join. You can think of merge joins as taking two sorted lists of values and merging them.
Suppose you have two piles of contractor information. One pile contains the master contract that each contractor has
signed, and the second contains descriptions of each project that the contractor has worked on. Both piles are sorted
by contractor name. You need to merge the two piles of paperwork so each contractor's master contract is placed
with all the project descriptions for that contractor. You basically need to make just one pass through each stack.

Going back to the employees and departments query, our pseudocode would look something like this:
GET one dept row and one empl row
DO (until one input is empty);
IF dept_no values are equal
Return values from both rows
GET next employee row
ELSE IF empl.dept_no < dept.dept_no
GET next employee row
ELSE GET next department row

The query optimizer usually chooses the merge join strategy when both inputs are already sorted on the join column.
In some cases, even if a sort must be done on one of the inputs, the query optimizer might decide that it is faster to
sort one input and then do a merge join than to use any of the other available strategies. If the inputs are both already
sorted, little I/O is required to process a merge join if the join is one to many. A many-to-many merge join uses a
temporary table to store rows instead of discarding them. If there are duplicate values from each input, one of the
inputs must rewind to the start of the duplicates as each duplicate from the other input is processed.

Hash Joins
Hash joins, also introduced in SQL Server 7, can be used when no useful index exists on the join column in either
input. Hashing is commonly used in searching algorithms (as discussed in detail in Donald Knuth's classic The Art of
Computer Programming: Sorting and Searching). We'll look at a simplified definition here. Hashing lets you
determine whether a particular data item matches an already existing value by dividing the existing data into groups
based on some property. You put all the data with the same value in what we can call a hash bucket. Then, to see if
a new value has a match in existing data, you simply look in the bucket for the right value.

For example, suppose you need to keep all your wrenches organized in your workshop so that you can find the right
size and type when you need it. You can organize them by their size property and put all half-inch wrenches (of any
type) together, all 10-mm wrenches together, and so on. When you need to find a 15-mm box wrench, you simply go
to the 15-mm drawer to find one (or see whether the last person who borrowed it has returned it). You don't have to
look through all the wrenches, only the 15-mm ones, which greatly reduces your searching time. Alternatively, you can
organize your wrenches by type and have all the box wrenches in one drawer, all the socket wrenches in another
drawer, and so on.

Hashing data in a table is not quite so straightforward. You need a property by which to divide the data into sets of
manageable size with a relatively equal membership. For integers, the hashing property might be something as simple
as applying the modulo operator to each existing value. If you decide to hash on the operation modulo 13, every value
is put in a hash bucket based on its remainder after dividing by 13. This means you need 13 buckets because the
possible remainders are the integers 0 through 12. In real systems, the actual hash functions are substantially more
complex; coming up with a good hash function is an art as well as a science.

When you use hashing to join two inputs, SQL Server uses one input, called the build input, to actually build the
hash buckets. Then, one row at a time, it inspects the other input and tries to find a match in the existing buckets. The
second input is called the probe input. Suppose we use modulo 3 as the hash function and join the two inputs shown
in Figure 15-6. The rows not matching the SARGs are not shown, so the two inputs are reduced in size quite a bit.
This document is created with the unregistered version of CHM2PDF Pilot

Here's the query:

SELECT Name, Manager
FROM employee empl JOIN department dept
ON dept.dept_no=empl.dept_no
WHERE <SARGs>

Figure 15-6. Two inputs to a hash join operation.

When processing hash joins, SQL Server tries to use the smaller input as the build input, so in this case we'll assume
that the hash buckets have been built based on the department table values. The buckets are actually stored as linked
lists, in which each entry contains only the columns from the build input that are needed. In this case, all we have to
keep track of is the department number and manager name. The collection of these linked lists is called the hash table
. Our hash table is shown in Figure 15-7. Our pseudocode looks something like this:
Allocate an Empty Hash Table
For Each Row in the Build Input
Determine the hash bucket by applying the hash function
Insert the relevant fields of the record into the bucket
For Each Record in the Probe Input
Determine the hash bucket
Scan the hash bucket for matches
Output matches
Deallocate the Hash Table

Applying this algorithm to our two inputs, we get these results:

Name Manager
-------- -------
Pike Laszlo
Delaney Graeffe
Moran Aebi
Dwyer Laszlo
Karaszi Soukup
This document is created with the unregistered version of CHM2PDF Pilot

Figure 15-7. A hash table built from the smaller (build) input.

Hashing Variations
Although you can use hashing effectively when there are no useful indexes on your tables, the query optimizer might
not always choose it as the join strategy because of its high cost in terms of memory required. Ideally, the entire hash
table should fit into memory; if it doesn't, SQL Server must split both inputs into partitions, each containing a set of
buckets, and write those partitions out to disk. As each partition (with a particular set of hash buckets) is needed, it is
brought into memory. This increases the amount of work required in terms of I/O and general processing time. In the
academic literature, you might see this type of hashing operation referred to as a Grace hash; the name comes from
the project for which this technique was developed. SQL Server has another alternative, somewhere in between
keeping everything in memory and writing everything out to disk. Partitions are only written out as needed, so you
might have some partitions in memory and some out on disk.

To effectively use hashing, SQL Server must be able to make a good estimate of the size of the two inputs and
choose the smaller one as the build input. Sometimes, during execution SQL Server will discover that the build input is
actually the larger of the two. At that point, it can actually switch the roles of the build and probe input midstream, in a
process called role reversal. Determining which input is smaller isn't too hard if there are no restrictive SARGs, as in
this query:
SELECT Name, Manager
FROM employee empl JOIN department dept
ON dept.dept_no=empl.dept_no

SQL Server knows the sizes of the inputs because sysindexes keeps track of the size of each table. However,
consider the case in which additional conditions are added to the query, as in this example:
WHERE empl.phone LIKE '425%'

It would help the query optimizer to know what percentage of the employees have phone numbers starting with 425.
This is a case in which column statistics can be helpful. Statistics can tell the query optimizer the estimated number of
rows that will meet a given condition, even if there is no actual index structure to be used.
NOTE
Hash and merge join can be used only if the join is an equijoin—that is, if the join predicate compares columns from
the two inputs for an equality match. If the join is not based on an equality, as in the example below, a nested loops
join is the only possible strategy:
WHERE table1.col1 BETWEEN table2.col2 AND table2.col3

The hashing and merging strategies can be much more memory intensive than the nested loops strategy, and because
the query optimizer takes into account all available resources, you might notice that a system with little available
memory might not use the hash and merge joins as often as systems with plenty of memory. Personal SQL Server
installations are particularly vulnerable to memory pressure because you're more likely to have many competing
This document is created with the unregistered version of CHM2PDF Pilot

applications running simultaneously. In most situations, there is limited memory for a desktop installation, so the query
optimizer is much more pessimistic when it decides to use one of these two strategies.

Multiple-Table Joins
According to some folklore, SQL Server "does not optimize" joins of more than four tables. There was some truth to
this in earlier versions of SQL Server, but in SQL Server 2000 optimization doesn't happen in the same way at all.
There are so many different possible permutations of join strategies and index usage that the query optimizer goes
through multiple optimization passes, each considering a larger set of the possible plans. On each pass, it keeps track
of the best plan so far and uses a cost estimate to reduce the search in succeeding passes. If your query contains
many tables, the query optimizer is likely to interrupt the last pass before completion because the last pass is quite
exhaustive. The cheapest plan found will then be used.

Other Processing Strategies

Although our discussion of the query optimizer has dealt mainly with the choice of indexes and join strategies, the
optimizer makes other decisions as well. It must decide how to process queries with GROUP BY, DISTINCT, and
UNION, and it must decide how updates should be handled—either row at a time or table at a time. We looked at
update strategies in Chapter 9. Here, I'll briefly discuss GROUP BY, DISTINCT, and UNION.

GROUP BY Operations
Prior to SQL Server 7, a GROUP BY operation was processed in only one way: SQL Server sorted the data (after
applying any SARGs) and then formed the groups from the sorted data. SQL programmers noticed that queries
involving grouping returned results in sorted order, although this was merely a by-product of the internal grouping
mechanism. In SQL Server 2000, the query optimizer can choose to use hashing to organize the data into groups and
compute aggregates. The pseudocode looks like this:
For Each Record in the Input
Determine the hash bucket
Scan the hash bucket for matches
If a match exists
Aggregate the new record into the old
Drop the new record
Otherwise
Insert the record into the bucket
Scan and Output the Hash Table

If the query optimizer chooses to use hashing to process the GROUP BY, the data does not come back in any
predictable order. The order actually depends on the order in which records appear in the hash buckets, but without
knowing the specific hash function, you can't predict this order. If you use a lot of GROUP BY queries, you might be
surprised that sometimes the results come back sorted and sometimes they don't. If you absolutely need your data in a
particular order, you should use ORDER BY in your query.
NOTE
If a database has its compatibility level flag set to 60 or 65, the query processor automatically includes an ORDER
BY in the query. This allows you to mimic the behavior of older SQL Server versions.

DISTINCT Operations
As with GROUP BY queries, the only way to remove duplicates before SQL Server 7 was to first sort the data. In
SQL Server 2000, the query optimizer can choose to use hashing to return only the distinct result rows. The algorithm
is much like the algorithm for hashing with GROUP BY, except that an aggregate does not need to be maintained:
This document is created with the unregistered version of CHM2PDF Pilot

For Each Record in the Input

Determine the hash bucket
Scan the hash bucket for matches
If a match exists
Drop the new record
Otherwise
Insert the record into the bucket
and return the record as output

UNION Operations
The UNION operator in SQL Server has two variations for combining the result sets of two separate queries into a
single result set. If you specify UNION ALL, SQL Server returns all rows from both inputs as the final result. If you
don't specify ALL, SQL Server removes any duplicates before returning the results. For performance reasons, you
should carefully evaluate your data, and if you know that the inputs are not overlapping, with no chance of duplicates,
you should specify the ALL keyword to eliminate the overhead of removing duplicates. If SQL Server has to find the
unique rows, it can do so in three ways: it can use hashing to remove the duplicates in a UNION, just as it does for
DISTINCT; it can sort the inputs and then merge them; or it can concatenate the inputs and then sort them to remove
the duplicates.

For GROUP BY, DISTINCT, and UNION operations, your best bet is to let the query optimizer decide whether to
use hashing or sorting and merging. If you suspect that the query optimizer might not have made the best choice, you
can use query hints to force SQL Server to use a particular processing strategy and compare the performance with
and without the hints. In most cases, the query optimizer will have made the best choice after all. I'll tell you about
query hints in Chapter 16.

Maintaining Statistics
As you've seen, the query optimizer uses statistics on both indexed and unindexed columns to determine the most
appropriate processing strategy for any particular query. Whenever an index is created on a table containing data (as
opposed to an empty table), the query optimizer collects statistics and stores them for that index in
sysindexes.statblob. By default, the statistics are automatically updated whenever the query optimizer determines that
they are out-of-date. In addition, the query optimizer generates statistics on columns used in SARGs when it needs
them, and it updates them when needed.

As you've seen, the histogram for statistics on an index key or statistics on an unindexed column is a set of up to 200
values for that column. All of the values or a sampling of the values in a given column are sorted; the ordered
sequence is divided into up to 199 intervals so that the maximum statistically significant information is captured. In
general, these intervals are of nonequal size. SQL Server 2000 builds the histogram from the sorted set of column
values in three passes. In the first pass, up to 200 values of RANGE_HI_KEY, EQ_ROWS, RANGE_ROWS, and
DISTINCT_RANGE_ROWS are collected. Each additional key or column value is processed in the second pass;
the value is either included in the last range (this is possible because the values are sorted) or a new range is created.
If a new range is created, one pair of existing, adjacent ranges is collapsed into a single range so that the number of
ranges (or steps) does not increase. The collapsed ranges are selected by considering the density information so that
two neighboring ranges with the closest densities are collapsed in order to minimize information loss. In the third pass,
more ranges might be collapsed if they have close densities. Therefore, even if the key column has more than 200
unique values, the histogram might have fewer than 200 steps.

You can specify the sampling interval when the indexes or statistics are initially created or at a later time using the
statistics management tools. You can specify the FULLSCAN option, which means that all existing column values are
sorted, or you can specify a percentage of the existing data values. By default, SQL Server uses a computed
percentage that is a logarithmic function of the size of the table. If a table is less than 8 MB in size, a FULLSCAN is
always done. When sampling, SQL Server randomly selects pages from the table by following the IAM chain. Once a
particular page has been selected, all values from that page are used in the sample. Since disk I/O is the main cost of
This document is created with the unregistered version of CHM2PDF Pilot

maintaining statistics, using all values from every page read minimizes that cost.

Statistics on nonindexed columns are built in exactly the same way as statistics on indexes. The system procedure
sp_helpstats shows column statistics, in much the same way that sp_helpindex shows indexes on a table. If the
statistics are created automatically, their names are very distinctive, as shown in this example:
EXEC sp_helpstats authors

RESULTS:

statistics_name statistics_keys
------------------------------- ---------------------------------
_WA_Sys_au_fname_75D7831F au_fname
_WA_Sys_state_75D7831F state
city_stats city

We have three sets of statistics. The set of explicitly generated statistics is city_stats. The system-generated statistics
are _WA_Sys_au_fname_75D7831F and _WA_Sys_state_75D7831F. To avoid long-term maintenance of unused
statistics, SQL Server ages out the statistics that are automatically created on unindexed columns. After it updates the
column statistics more than a certain number of times, the statistics are dropped rather than updated again. The
StatVersion column of the sysindexes table shows how many times the statistics have been updated. Once
StatVersion exceeds an internally generated threshold, the statistics are dropped. If they are needed in the future,
they can be created again. There is no substantial cost difference between creating statistics and updating them.
Statistics that are explicitly created (such as city_stats in the preceding example) are not automatically dropped.

Statistics are automatically updated when the query optimizer determines that they are out-of-date. This basically
means that sufficient data modification operations have occurred since the last time they were updated to minimize
their usefulness. The number of operations is tied to the size of the table and is usually something like 500 + 0.2 *
(number of rows in the table). This means that the table must have at least 500 modification operations before
statistics are updated during query optimization; for large tables, this threshold can be much larger. In this case, when I
say "operation" I mean a single row modification. A single UPDATE statement that changes 1000 rows will be
counted as 1000 modification operations. You can actually see the number of modification operations in the
sysindexes table, in the rowmodctr column. The number of rows affected by every INSERT, UPDATE, or DELETE
operation on a table is reflected in the rowmodctr column for the row representing the table itself. Remember that this
will be the sysindexes row that has an indid value of either 0 (if the table is a heap) or 1 (if the table has a clustered
index). Whenever statistics are updated, either manually or automatically, the rowmodctr value is reset to 0.

You can also manually force statistics to be updated in one of two ways. You can run the UPDATE STATISTICS
command on a table or on one specific index or column statistics, or you can also execute the procedure
sp_updatestats, which runs UPDATE STATISTICS against all user-defined tables in the current database.

You can create statistics on unindexed columns using the CREATE STATISTICS command or by executing
sp_createstats, which creates single-column statistics for all eligible columns for all user tables in the current
database. This includes all columns except computed columns and columns of the ntext, text, or image datatypes,
and columns that already have statistics or are the first column of an index.

Every database is created with the database options auto create statistics and auto update statistics set to true,
but you can turn either one off. You can also turn off automatic updating of statistics for a specific table in one of two
ways:
 sp_autostats This procedure sets or unsets a flag for a table to indicate that statistics should or should not be
updated automatically. You can also use this procedure with only the table name to find out whether the table
is set to automatically have its index statistics updated.
 UPDATE STATISTICS In addition to updating the statistics, the option WITH NORECOMPUTE
indicates that the statistics should not be automatically recomputed in the future. Running UPDATE
STATISTICS again without the WITH NORECOMPUTE option enables automatic updates.
This document is created with the unregistered version of CHM2PDF Pilot

However, setting the database option auto update statistics to FALSE overrides any individual table settings. In
other words, no automatic updating of statistics takes place. This is not a recommended practice unless thorough
testing has shown you that you don't need the automatic updates or that the performance overhead is more than you
can afford.

The Query Plan

The end product of the compilation phase of query processing is a query plan, which, with some exceptions, is put
into the procedure cache. Cheap ad hoc SQL plans don't really go in the cache because it's not a good idea to flood
the cache with plans that are unlikely to be reused. And ad hoc SQL is the least likely type of query to have its plan
reused. If the statement is already so cheap to compile (less than 1/100 second), there is little to be gained by storing
the plan in the cache.

Figure 15-8 shows the flow of compilation and execution. If a plan is not in cache initially, we must create one. This is
shown on the left side of the flowchart. You can also see that after the plan has been put in cache, when it's time to
execute it, SQL Server checks to see whether anything has changed and whether the plan must be recompiled. Even
though it could be only microseconds separating compilation from execution, somebody could have executed a Data
Definition Language (DDL) statement that added an index to a crucial table. This is admittedly unlikely, but it is
possible, so SQL Server must account for it. There are a few things that will cause SQL Server to recompile a stored
plan. Metadata changes such as adding or dropping indexes is the most likely reason. You need to make sure that the
plan used reflects the current state of the indexes.

Another cause of recompilation is statistical changes. SQL Server keeps quite a bit of histogram information on the
data that it processes. If the data distribution changes a lot, you'll likely need a different query plan to get the most
efficient execution. If SQL Server has updated the statistics on an index used in the plan, a subsequent use of that plan
invokes recompilation based on new distribution information.

Figure 15-8. The flow of query compilation and execution.

Keep in mind that a change of actual parameter will not cause a plan to be recompiled, nor will environmental
changes such as the amount of memory available or the amount of needed data that is already in cache. I'll give you
more details about some special issues with stored procedure recompilation in the next section.

Execution is pretty straightforward, and if all you had were very simple queries such as "insert one row" or a singleton
select from a table with a unique index, the processing would be very simple. But many queries require large amounts
This document is created with the unregistered version of CHM2PDF Pilot

of memory to run efficiently, or at least could benefit from that memory. Starting with SQL Server 7, the SQL Server
developers added the ability to use large amounts of memory for individual queries.

Another problem came up at this point. Once you start allowing queries to use very large amounts of memory, you
have to decide how to arbitrate this memory use among many queries that might need that memory. SQL Server
handles this as follows. When a query plan is optimized, the optimizer determines two pieces of information about
memory use for that query: first, it picks a minimum amount of memory that the query needs in order to run effectively,
and that value is stored with the query plan. It also determines a maximum amount of memory that the query could
benefit from. For example, it doesn't help to give a sort operation 2 GB of memory if the entire table you're sorting fits
into 100 MB. You really just want the 100 MB, and so that would be the value for the maximum amount of useful
memory that's stored with the query plan.

When SQL Server goes to execute the plan, the plan gets passed to a routine called the memory grant scheduler. If
the query it is looking at doesn't have a sort or a hash operation as part of the plan, SQL Server knows it doesn't use
large amounts of memory. In this case, there is no waiting in the memory grant scheduler at all. The plan can be run
immediately, so a typical transaction processing request bypasses this mechanism completely.

If there are very large queries, it's best to run a few of them at a time and let them get more of the memory they need.
If at least half of the maximum memory requirement of the query is available, the query will run immediately with
whatever memory it can get. Otherwise, the query will wait in the memory grant queue until half of the maximum
memory requirement is available. At the same time, the SQL Server will never give out more than half of the remaining
query memory to any one query. If you find that you're getting lots of memory timeout errors (error number 8645),
this means your queries were waiting too long in the memory grant scheduler. By default, a query will wait only for an
amount of time that is proportional to its estimated cost to run; at this time, that value is about 25 times the estimated
number of seconds. With the default value, no query will wait less than 25 seconds because the query cost is rounded
up to 1 if it is less than 1 second. If the time limit is exceeded, error 8645 is generated.

Once the scheduler allows memory to be allocated for a query's execution, SQL Server performs a step called
"opening the plan," which starts the actual execution. From there, the plan runs to completion. If your query is using
the default result set model, the plan will actually just run until it has produced all of its rows and they've all been sent
back to the client. If you're using a cursor model, the processing is a bit different. Each client request is for just one
block of rows, not all of them. After each block is sent back to the client, SQL Server must wait for the client to
request the next block. While it's waiting, the entire plan is quiesced—that is, some of the locks are released, some
resources are given up, and some positioning information is squirreled away. This information allows SQL Server to
get back to where it was when the next block of rows is requested, and execution can continue.

I mentioned earlier that compilation and execution are two distinct phases of query processing, so one of the things
the optimizer does is optimize based on a fairly stable state. SQL Server might recompile a statement depending on
certain criteria being met, in which case the state isn't permanently stable but it must be something that you know
doesn't change too often. If the optimizer were to use information that often changes dramatically, such as the number
of concurrent processes or the number of locks held, you'd constantly be recompiling things. And compilation,
especially in SQL Server 2000, tends to be pretty slow. For example, you might have a SQL statement that runs in
1/100 second, but it takes a ½ second to compile. It would be preferable to compile the statement once and let
people execute it thousands or millions of times without repaying the compilation cost on every statement.
This document is created with the unregistered version of CHM2PDF Pilot

34

The Procedure Cache

I've mentioned SQL Server's procedure cache several times. It's important to understand that the procedure cache in
SQL Server 2000 is not actually a separate area of memory. Releases prior to SQL Server 7 had two effective
configuration values to control the size of the procedure cache: one specifying a fixed size for SQL Server's total
usable memory and one specifying a percentage of that memory (after fixed needs were satisfied) to be used
exclusively for storing query plans. Also, in previous versions query plans for ad hoc SQL statements were never
stored in the cache, only the plans for stored procedures. In SQL Server 2000, the total size of memory is by default
dynamic and the space used for query plans is also very fluid.

One of the first things SQL Server determines when processing a query is whether the query is both ad hoc and
cheap to compile; if so, SQL Server won't bother caching it at all. (In the next section, we'll look at what "ad hoc"
actually means and compare it to the alternative types of queries: autoparameterized, prepared, and stored
procedures.) It's not a good idea to flood the cache with things that probably won't be reused very often. It is cheaper
to simply recompile if necessary. If the query is not ad hoc or is not cheap to compile, SQL Server will get some
memory from the buffer cache. There is only one source of memory for 99 percent of the server's needs, and that is
the buffer cache itself. In a few cases, SQL Server will go out and allocate large chunks of memory directly from the
operating system, but those cases are extremely rare. For everything else, the management is centralized.

Plans are saved in cache along with a cost factor that reflects the cost of actually creating the plan by compiling the
query. If the plan is ad hoc, SQL Server will set its cost to 0. The value 0 means that the plan is immediately available
to be kicked out of the procedure cache. You're taking a chance that the plan might be reused, but the probability is
low. If there's memory pressure on the system, plans for ad hoc statements should be the first to go. If the plan is not
ad hoc, SQL Server sets the cost to the actual cost of compiling the query. These costs are basically in units of disk
I/O. When a data page is read off the disk, it has a cost of 1 I/O. When this plan was built, information was read off
the disk, including statistics and the text of the query itself. SQL did additional processing and that processing work is
normalized to the cost of an I/O. Once the cost is computed, the plan is put into the cache. Figure 15-9 shows the
flow of costing a plan and placing it in cache.

If another query comes along that can reuse that plan, SQL Server again determines what type of plan it is. If it's an
ad hoc plan, SQL Server increments the cost by 1. So if ad hoc plans are really being reused, they'll stay in the cache
for a little while longer as their cost factor increases. If the plan is reused frequently, the cost will keep getting bumped
up by 1 until it reaches its actual creation cost—and that's as high as it will ever be set. But if it's reused a lot—if the
same user or other users keep resubmitting the exact same SQL text—the plan will remain in cache.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 15-9. Costing a query plan and placing it in cache.

If the query is not ad hoc, which means that it's a stored procedure, a prepared query, or an autoparameterized
query, the cost is set back up to the creation cost every time the plan is reused. As long as the plan is reused, it will
stay in cache. Even if it's not used for a while, depending on how expensive it was to create initially, it can remain in
cache for a fair amount of time. Figure 15-10 shows the flow of retrieving a plan from cache, and adjusting the cost.

Figure 15-10. Retrieving a plan from cache.

The lazywriter is the mechanism that actually manages the cost of plans and is responsible for removing plans from
cache when necessary. The lazywriter is actually part of the storage engine, but since it's so important to the query
processing mechanism that it needs to be mentioned here. The lazywriter uses the same mechanism for managing
memory used by query plans that it does for managing data and index pages, because plans are stored in the normal
buffer cache. The lazywriter looks through all the buffer headers in the system. If there's very little memory pressure
on the system, it looks very slowly; if memory pressure grows and the hit rates on the cached objects start going up,
the lazywriter starts running more frequently. As the lazywriter runs, it looks at each buffer header and then at the
current cost for the page that the buffer is holding. If the cost is 0, which means it hasn't been touched since the
lazywriter's last pass through the cache, the lazywriter will release that page to get some free memory in the system to
use for page I/O or for other plans. In addition, if the buffer contains a procedure plan, the lazywriter will call the SQL
manager, which does some cleanup. Finally, the buffer is put on the free list for reuse.

If the cost associated with a buffer is greater than 0, the lazywriter decrements the cost and continues inspecting other
buffers. The cost then actually indicates for how many cycles of the lazywriter something will stay in the cache without
reuse before it's removed. This algorithm doesn't differentiate between query plans in cache and data or index pages
in cache, with the exception of the step that calls the SQL manager if the object is a stored procedure. The lazywriter
doesn't really know anything about an object being a query plan, and the algorithm balances the use of a cache for
disk I/O versus the use of cache for a procedure plan. Figure 15-11 shows the flow of the lazywriter's processing of
the cache.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 15-11. The flow of the lazywriter's processing of plans in cache.

You'll find that if you have something that is expensive to compile, even if it is not touched for a long time, it will stay
in cache because its initial cost is so high. Anything that is reused frequently will sit in cache for a long time because its
cost is also reset whenever it's used, and the lazywriter will never see it go to 0.
This document is created with the unregistered version of CHM2PDF Pilot

34

Using Stored Procedures and

Caching Mechanisms
Stored procedures are the most cost-effective mechanism for executing SQL commands from your applications, and
I recommend that you use them whenever possible rather than passing ad hoc SQL statements from your
applications. Chapters 3 and 11 discussed the efficiencies that stored procedures bring in terms of not requiring
compilation of an execution plan for each execution. Plans can be reused and stay cached and available for
subsequent use. Reusing a saved plan can provide a major performance advantage. The query optimizer might
evaluate thousands of possible plans, so compilation and optimization time can end up being a more significant
percentage of total execution time. It is critical that you understand when recompilation takes place and try to avoid it
if at all possible.

SQL Server can save on recompilation using four mechanisms to make plan caching accessible in a wider set of
scenarios than just using stored procedures:
 Ad hoc caching
 Autoparameterization
 The sp_executesql procedure
 The prepare and execute method
NOTE
The information here on caching query plans was adapted from a preliminary copy of a whitepaper by Peter
Scharlock.

Ad Hoc Caching
SQL Server caches the plans from ad hoc queries, and if a subsequent batch matches exactly, the cached plan is
used. This feature requires no extra work to use, but it is limited to exact textual matches. For example, if the
following three queries are submitted in the Northwind database, the first and third queries will use the same plan but
the second one will need to generate a new plan:
SELECT * FROM orders WHERE customerID = 'HANAR'
SELECT * FROM orders WHERE customerID = 'CHOPS'
SELECT * FROM orders WHERE customerID = 'HANAR'

Autoparameterization
For simple queries, SQL Server guesses which constants might really be parameters and attempts to treat them as
parameters. If this is successful, subsequent queries that follow the same basic template can use the same plan. The
following four templates can be used for autoparameterization. Note that key-expression is an expression involving
only column names, constants, AND operators, and comparison operators

(<, >, =, <=, >=, and <>).

INSERT table VALUES ({constant | NULL | DEFAULT}, ...)

DELETE table WHERE key-expression
UPDATE table SET colname = constant WHERE key-expression
SELECT {* | column-list} FROM table WHERE key-expression ORDER BY column-list
This document is created with the unregistered version of CHM2PDF Pilot

For example, these two queries will use the same plan:
SELECT firstname, lastname, title FROM employees WHERE employeeID = 6
SELECT firstname, lastname, title FROM employees WHERE employeeID = 2

Internally, SQL Server parameterizes these queries as follows:

SELECT firstname, lastname, title FROM employees WHERE employeeID = @p

SQL Server can allow other queries of the same template to use the same plan only if the template is safe. A
template is safe if the plan selected will not change even if the actual parameters change. This ensures that
autoparameterization won't degrade a query's performance.

The SQL Server query processor is much more conservative about deciding whether a template is safe than an
application can be. SQL Server guesses which values are really parameters, whereas your application should actually
know. Rather than rely on autoparameterization, you should use one of the following two mechanisms to mark
parameters when they are known.

The sp_executesql Procedure

The stored procedure sp_executesql is halfway between ad hoc caching and stored procedures. Using
sp_executesql requires that you identify the parameters but doesn't require all the persistent object management
needed for stored procedures.

Here's the general syntax for the procedure:

sp_executesql @batch_text, @batch_parameter_definitions,

param1,...paramN

Repeated calls with the same batch_text use the cached plan, with the new parameter values specified. The same
cached plan is used in all the following cases:
EXEC sp_executesql N'SELECT firstname, lastname, title FROM employees
WHERE employeeID = @p', N'@p tinyint', 6
EXEC sp_executesql N'SELECT firstname, lastname, title FROM employees
WHERE employeeID = @p', N'@p tinyint', 2
EXEC sp_executesql N'SELECT firstname, lastname, title FROM employees
WHERE employeeID = @p', N'@p tinyint', 6

ODBC and OLE DB expose this functionality via SQLExecDirect and ICommandWithParameters. The ODBC
and OLE DB documentation provide more details.

The Prepare and Execute Method

This last mechanism is like sp_executesql in that parameters to the batch are identified by the application, but there
are some key differences. The prepare and execute method does not require the full text of the batch to be sent at
each execution. Rather, the full text is sent once at prepare time; a handle that can be used to invoke the batch at
execute time is returned. ODBC and OLE DB expose this functionality via SQLPrepare/SQLExecute and
ICommandPrepare. You can also use this mechanism via ODBC and OLE DB when cursors are involved. When
you use these functions, SQL Server is informed that this batch is meant to be used repeatedly.
This document is created with the unregistered version of CHM2PDF Pilot

Sharing Cached Plans

To allow users to share plans and thus maximize the effectiveness of the caching mechanisms, all users should execute
in the same environment. Don't change SET options or database settings in the middle of an application or connection.

For all of the caching mechanisms, reusing a cached plan avoids recompilation and optimization. This saves
compilation time, but it means that the same plan is used regardless of the particular parameter values passed in. If the
optimal plan for a given parameter value is not the same as the cached plan, the optimal execution time will not be
achieved. For this reason, SQL Server is very conservative about autoparameterization. When an application uses
sp_executesql, prepare and execute, or stored procedures, the application developer is responsible for determining
what should be parameterized. You should parameterize only constants whose range of values does not drastically
affect the optimization choices.

The SQL Server Performance Monitor includes a counter called SQL Server:SQL Statistics that has several counters
dealing with autoparameterization. You can monitor these counters to determine whether there are many unsafe or
failed autoparameterization attempts. If these numbers are high, you can inspect your applications for situations in
which the application can take responsibility for explicitly marking the parameters.

Examining the Plan Cache

The system table syscacheobjects keeps track of the compiled objects in cache at any time. Note that this table
exists only in the master database and is really a pseudotable. It takes no space on disk and is materialized only when
someone writes a query to access it. Table 15-2 lists some of the more useful columns in the syscacheobjects table.

Table 15-2. Useful columns in the syscacheobjects table.

Column Name Description
The bucket ID for this plan in an internal hash table; the
bucket ID helps SQL Server locate the plan more
bucketid
quickly. Two rows with the same bucket ID refer to the
same object (for example, the same procedure or trigger).
cacheobjtype Type of object in the cache.
objtype Type of object.
One of the main keys used for looking up an object in the
cache. This is the object ID stored in sysobjects for
objid database objects (procedures, views, triggers, and so on).
For cache objects such as ad hoc or prepared SQL,
objid is an internally generated value.
dbid Database ID in which the cache object was compiled.
The creator of the plan (for ad hoc query plans and
uid
prepared plans).
Number of other cache objects that reference this cache
refcounts
object.
Number of times this cache object has been used since
usecounts
inception.
pagesused Number of memory pages consumed by the cache object.
SET option settings that affect a compiled plan. Changes
setopts to values in this column indicate that users have modified
SET options.
This document is created with the unregistered version of CHM2PDF Pilot

Language ID of the connection that created the cache

langid
object.
Date format of the connection that created the cache
dateformat
object.
Procedure name or first 128 characters of the batch
sql
submitted.

Cacheobjtype refers to the type of plan. It can have one of the following values:
 Compiled Plan The actual plan generated during compilation and optimization. A compiled plan can be
shared by sessions each running the same procedure or query.
 Executable Plan The execution environment that runs the compiled plan. Each concurrent invocation of the
same compiled plan will have its own executable plan. Caching this environment information saves setup
costs, which can be significant for a frequently run query. All executable plans must have an associated
compiled plan with the same bucketid, but not all compiled plans will have an associated executable plan
saved.
 Parse Tree The internal form of a query prior to compilation and optimization.
 Cursor Parse Tree The parse tree for a cursor's query.

Objtype refers to the type of object whose plan is cached. It can have one of the following values:
 Proc For both stored procedures and inline functions.
 Prepared For queries submitted using sp_executesql and using the prepare and execute method and for
queries that SQL Server has decided to autoparamaterize.
 Ad hoc query For queries that don't fall into any other category (as discussed above).
 ReplProc For replication agents.
 Trigger A type whose plans are very similar to procedure or inline function plans.
 View A type for which you typically see only a parse tree, not a plan. You see this type not only when a view
is accessed, but also when a non-inline function is encountered. The function does not have its own separate
plan, but is expanded as part of whatever query it is part of.
 Table A type for which a user or system table will generate a parse tree object if it has computed columns.
 Default, Check, Rule Types that are expanded into parse trees and compiled as part of the query in which
they are activated.

Two other columns, refcount and usecount, indicate how the plans are being used. The refcount value indicates how
many times the plan has been referenced. An executable plan in cache will always have a refcount of 1. An
executable plan might actually have a refcount greater than 1 if the plan is currently being executed, but that will not
show up in the syscacheobjects table. The refcount for compiled plans is 1 + the number of execution plans, either in
cache or in use. The usecount value refers to the number of times a cached object is used. Each time the plan is
looked up and found, the usecount is incremented. If you see a value of 2 for usecount after the first execution of a
procedure or query, that is because the plan is accessed once when it is created and stored in cache, and once again
when it is retrieved from cache and executed.

Multiple Plans in Cache

SQL Server will try to limit the number of plans for a query or a procedure. Because plans are reentrant, this is easy
to accomplish. You should be aware of some situations that will cause multiple plans for the same procedure to be
saved in cache. The most likely situation is a difference in certain SET options, database options, or configuration
options. For example, a stored procedure that concatenates strings might compile the concatenation differently
depending on whether the option CONCAT_NULL_YIELDS_NULL is on or off or whether the corresponding
database option is true or false. If a user executes the procedure with the option on, that person will use a different
plan than if the option is off. The column setopts in the syscacheobjects table is a bitmap indicating which SET
options were in effect when the plan was placed in cache. If you start a different session with different SET options
enabled, SQL Server will always generate a new plan. The options included in the setopts field include:
This document is created with the unregistered version of CHM2PDF Pilot

ANSI_PADDING
FORCEPLAN
CONCAT_NULL_YIELDS_NULL
ANSI_WARNINGS
ANSI_NULLS
QUOTED_IDENTIFIER
ANSI_NULL_DFLT_ON
ANSI_NULL_DFLT_OFF

Two sessions that have set different languages or different dateformat values will also always generate separate
plans. SQL Server must then also keep the langid value and dateformat value in syscacheobjects so that these
values can be compared with future attempts to execute the same plan.

One other connection issue can affect whether a plan can be reused. If an owner name must be resolved implicitly, a
plan cannot be reused. For example, suppose user sue issues the following SELECT statement:
SELECT * FROM mytable

SQL Server will first try to resolve the object by looking for an object called mytable owned by sue, and if no such
object can be found, it will look for an object called mytable owned by the DBO. If user dan executes the exact
same query, the object can be resolved in a completely different way (to a table owned by dan), so sue and dan
could not share the plan generated for this query. However, the situation is different if sue issues this command:
SELECT * FROM dbo.mytable

Now there's no ambiguity. Anyone executing this exact query will always reference the same object. In the
syscacheobjects table, the column uid indicates the user ID for the connection in which the plan was generated. Only
another connection with the same user ID value can use the same plan. The one exception is if the user ID value is
recorded as _2 in syscacheobjects, which indicates that the query submitted does not depend on implicit name
resolution and can be shared among different users. This is the preferred method.
TIP
If your server has been running for a while, the syscacheobjects table can get quite large and awkward to work with.
If you want to run your own tests to determine what gets put in cache, and when plans are reused, you will probably
want to clear out the cache occasionally to keep its size manageable. This can be done with the command DBCC
FREEPROCCACHE, which removes all cached plans from memory. In addition, DBCC FLUSHPROCINDB(<
dbid>) can be used to clear all plans from one particular database. DO NOT use these commands on your
production servers as it could impact the performance of your running applications.

When to Use Stored Procedures and Other Caching

Mechanisms
Keep the following guidelines in mind when you are deciding whether to use stored procedures or one of the other
mechanisms:
 Stored procedures Use when multiple applications are executing batches in which the parameters are
known.
 Ad hoc caching Useful in limited scenarios. Don't design an application to use this.
 Autoparameterization Use for applications that cannot be easily modified. Don't design an application to
use this.
 The sp_executesql procedure Use when a single user might use the same batch multiple times and when the
parameters are known.
 The prepare and execute method Use when multiple users are executing batches in which the parameters
are known, or when a single user will definitely use the same batch multiple times.
This document is created with the unregistered version of CHM2PDF Pilot

Recompiling Stored Procedures

In spite of the benefits of the mechanisms just discussed, you should still use stored procedures whenever possible.
Besides giving you the benefits of precompilation, they minimize network traffic by reducing the text that needs to be
sent from your application to SQL Server, and they provide a security mechanism to control who can access data and
under what conditions. Stored procedures can be recompiled automatically or manually under various circumstances.
Chapter 11 discussed some of these issues.

You can actually watch automatic recompilation occurring by using SQL Profiler: Choose to trace the event in the
Stored Procedures category called SP:Recompile. If you also trace the event called SP:StmtStarting, you can see at
what point in the procedure it is being recompiled. In particular, you might notice that stored procedures can be
recompiled multiple times during their execution. For example, if you build a temporary table and later create an index
on that table, and then add data to the table, your stored procedure will be recompiled numerous times. One way to
avoid this is to include all data definition statements dealing with your temporary tables, as well as the insertion of rows
into the temporary tables, right at the beginning of the procedure. So if the procedure must be recompiled, it won't
happen more than once. Another way to prevent recompilations is to include the query hint KEEPPLAN in your
statements that access the temporary tables. I'll discuss query hints in Chapter 16.

Although we've been assuming that recompilation is something you will usually want to avoid, this is not always the
case. If you know that updated statistics can improve the query plan, or if you know that you have wildly different
possible parameter values, recompilation can be a good thing.

As you saw, the system table syscacheobjects keeps track of the compiled objects in cache at any time. This table is
accessible only by system administrators, but you can write your own stored procedures to provide information for
your own tuning and testing purposes. You can use a procedure called sp_procs_in_cache (which is on the
companion CD) to return a list of all procedure plans in cache and the number of times each plan occurs. The list is
organized by database. If you execute the procedure without passing a parameter, you see procedures in your current
database only. Alternatively, you can provide a specific database name or use the parameter all to indicate that you
want to see procedure plans in cache from all databases. If you want to pull different information out of
syscacheobjects, you can customize the procedure in any way you like. Remember to create the procedure in the
master database so that it can be called from anywhere. Also remember to grant appropriate permissions if anyone
other than a system administrator will be running it. The sp_procs_in_cache procedure shows both compiled plans
and executable plans.

Other Benefits of Stored Procedures

Beyond the significant performance and security advantages, stored procedures can provide a valuable level of
indirection between your applications and the database design. Suppose your application calls a procedure such as
get_customer_balance and expects a result set to be returned. If the underlying database is then changed, the
application can be totally unaffected and unaware of the change as long as the procedure also changes to return the
result set as expected. For example, if you decide to denormalize your database design to provide faster query
performance, you can change the stored procedure to respecify the query. If many applications call the procedure,
you can simply change the stored procedure once and never touch the application. In fact, a running application
doesn't even need to be restarted—it executes the new version of the stored procedure the next time it is called.

Another good reason to use stored procedures is to minimize round-trips on the network (that is, the conversational
TDS traffic between the client application and SQL Server for every batch and result set). If you will take different
actions based on data values, try to make those decisions directly in the procedure. Strictly speaking, you don't need
to use a stored procedure to do this—a batch also provides this benefit. Issue as many commands as possible in a
batch. You can try a simple test by inserting 100 rows first as a single batch and then as every insert in its own batch
(that is, 100 batches). You'll see a performance improvement of an order of magnitude even on a LAN, and on a
slow network the improvement will be stunning. While LAN speeds of 10 megabits per second (Mbps) are fast
This document is created with the unregistered version of CHM2PDF Pilot

enough that the network is generally not a significant bottleneck, speeds can be tremendously different on a slow
network. A modem operating at 28.8 kilobits per second (Kbps) is roughly 300 times slower than a LAN. Although
WAN speeds vary greatly, they typically can be 100 to 200 times slower than a LAN. The fast LAN performance
might hide inefficient network use. You can often see a prime example of this phenomenon when you use cursors.
Fetching each row individually might be tolerable on a LAN, but it is intolerable when you use dial-up lines.

If a stored procedure has multiple statements, by default SQL Server sends a message to the client application at the
completion of each statement to indicate the number of rows affected for each statement. This is known as a
DONE_IN_PROC message in TDS-speak. However, most applications do not need DONE_IN_PROC messages.
So if you are confident that your applications do not need these messages, you can disable them, which can greatly
improve performance on a slow network when there is otherwise little network traffic.

You can use the connection-specific option SET NOCOUNT ON to disable these messages for the application.
While they are disabled, you cannot use the ODBC SQLRowCount function or its OLE DB equivalent. However,
you can toggle NOCOUNT on and off as needed, and you can use select @@rowcount even when NOCOUNT is
on. You can also suppress the sending of DONE_IN_PROC messages by starting the server with trace flag 3640,
which lets you suppress what might be needless overhead without touching your application. However, some ODBC
applications depend on the DONE_IN_PROC message, so you should test your application before using trace flag
3640 in production—it can break applications that are written implicitly to expect that token.

You should keep an eye on the traffic between your client applications and the server. An application designed and
tuned for slow networks works great on a fast network, but the reverse is not true. If you use a higher-level
development tool that generates the SQL statements and issues commands on your behalf, it is especially important to
keep an eye on what's moving across the network. You can use the SQL Profiler to watch all traffic into the server.
Just to monitor network traffic, though, this might be overkill; a network sniffer such as Network Monitor (available
with Microsoft Windows NT Server, Microsoft Windows 2000 Server and Microsoft Systems Management Server)
can work better if you want only to monitor network traffic. Network Monitor is easy to use, and even a networking
neophyte can set it up quickly and with a few mouse clicks watch the traffic between machine pairs.
This document is created with the unregistered version of CHM2PDF Pilot

34

Execution
The final step in query processing is execution. I'm not going to talk about execution in any detail other than this short
paragraph. The execution engine takes the plan that the optimizer has produced and executes it. Along with the actual
execution, the execution engine schedules the threads for the processes to run on and provides interthread
communication.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
In this chapter, we looked at how SQL Server processes the queries that it receives from a client application. The
most powerful and complex component of query processing is the query optimizer. We looked at the information that
the optimizer uses to make its decisions about which indexes and which processing techniques should be used. Some
of the most important information available to the optimizer is the statistics. Statistics can be automatically generated
and maintained by SQL Server, and you can also manually update them.

We also looked at the caching of plans generated by the optimizer. SQL Server can cache and reuse plans not only
from stored procedures, but also from ad hoc and autoparameterized queries. Because generation of query plans can
be expensive, it is helpful to understand how and why query plans are reused and when they must be regenerated.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 16

Query Tuning
If you want to end up with a poorly performing application or a complete project failure, you should wait until the end
of the project to deal with performance concerns. If, however, you want your application to be the best it can be, you
must consider performance throughout the development cycle. In fact, you must consider performance before you
even write your first line of code.

Every chapter of this book has included some information on performance. If you've turned directly to this chapter in
hopes of finding the secret to improving your lackluster application, you'll be disappointed. I can offer only guidelines
for you to keep in mind, along with some pointers that refer back to earlier information or to other helpful materials.

Microsoft SQL Server systems can be brilliantly fast with well-designed, well-implemented applications. It can
support workloads of the type and size that no one dreamed possible back in 1988 when SQL Server first became
available. But with a poorly planned or poorly implemented system, SQL Server can perform horribly. Statements
like "SQL Server is slow" are not uncommon. Neither are such statements about other database products. Anytime
you hear this from someone who has deployed a production application, your first thought should be that the person
has dropped the ball somewhere along the line—or that SQL Server is unsuitable for the task at hand. (SQL Server
can handle most systems, but some are still beyond its reach.)

If this is the first chapter you've turned to, please stop and go back at least to Chapter 3. All of the chapters from
Chapter 3 to this one are relevant to performance issues. You might also revisit Chapter 13, which covers cursors,
and Chapter 14, which covers locking. A thorough understanding of cursors and locking is a prerequisite for
understanding the material in this chapter.

The big gains in query performance usually do not come from syntactic changes in the query but rather from a change
in the database design or in indexing—or from taking a completely different approach to the query. For example, you
might have to choose among the following approaches: writing a pretty complex query using a self-join or multilevel
correlated subquery, using a cursor, or creating a solution that involves temporary tables. (You saw some of these
techniques in Chapter 10.) Invariably, the only way you can determine which solution is best is to try all the queries.
You might be surprised by the results.

Rather than trying to learn a bunch of tricks to do up front, you should be much more interested in doing the basics
right. That is, you need to be sure that you've set up a good database structure—including perhaps some
denormalization based on your CRUD analysis (described below)—and that you've created in advance what appear
to be useful indexes. From there, you can test your queries and study the query plans generated for any queries that
seem problematic.
This document is created with the unregistered version of CHM2PDF Pilot

34

The Development Team

A software project's success depends on the experience and skill of the staff developing it. Your second SQL Server
development project will be better than your first, no matter how smart you are, so don't make your first project one
that will have thousands of concurrent users, manage tens of gigabytes of data, and replicate data to 10 other servers.
If you tackle such an extensive project your first time out, you'll probably fail (or at least finish late and over budget).
And although it's useful to have experience working with other systems and environments, you can't expect successful
SQL Server development on the first try.

The smartest companies start with a small, less-than-mission-critical system before moving many applications over to
SQL Server. If management will not allow you the luxury of working on a "practice" system, you should at least try to
augment your team with an experienced consultant or two. Look for someone who, at a minimum, is a Microsoft
Certified Systems Engineer (MCSE) or Microsoft Certified Solution Developer (MCSD) who has taken both basic
SQL Server exams—the Database Implementation and the Database Administration exams. In 1999, Microsoft
introduced a Microsoft Certified DBA (MCDBA) certification, which indicates a similar level of qualification. This
book, while it provides much useful information, can't replace other training. And skills such as database design, which
transcend SQL Server specifically, are essential for a project's success.
This document is created with the unregistered version of CHM2PDF Pilot

34

Application and Database Design

The biggest performance gains come from changes to the application and database design. You might change your
configuration settings and add heftier hardware and be thrilled when performance doubles, but changes to the
application can often result in even larger performance increases. There are as many approaches to software
development as there are pages in this book. No single approach is the right approach—yours must be tailored to the
size of the project, your team, and the skill level of the team members.

Take a look at the following list of suggestions for planning and implementing good performance in your system. I'll
explain most of these items in detail in this chapter and in Chapter 17. (Chapter 4 discusses hardware.)
 Develop expertise on your development team.
 Understand that there is no substitute for solid application and database design.
 State performance requirements for peak, not average, use.
 Consider perceived response time for interactive systems.
 Prototype, benchmark, and test throughout the development cycle.
 Create useful indexes.
 Choose appropriate hardware.
 Use cursors judiciously.
 Use stored procedures almost all of the time.
 Minimize network round-trips.
 Understand concurrency and consistency tradeoffs.
 Analyze and resolve locking (blocking) problems.
 Analyze and resolve deadlock problems.
 Monitor and tune queries using SQL Profiler.
 Monitor Microsoft Windows 2000 system using System Monitor.
 Review and adjust operating system settings.
 Review and adjust SQL Server configuration settings.
 Make only one change at a time, and measure its effect.
 Do periodic database maintenance.

Normalize Your Database

I assume that if you're reading this book, you understand the concept of normalization and terms such as third
normal form. (If you don't, see Candace Fleming and Barbara von Halle's Handbook of Relational Database
Design and Michael Hernandez's Database Design for Mere Mortals. A plethora of other books about database
design and normalization are also available.)

A normalized database eliminates functional dependencies in the data so that updating the database is easy and
efficient. But querying from that database might require a lot of joins between tables, so common sense comes into
play. If the most important and time-critical function your system must perform is fast querying, it often makes sense to
back off from a normalized design in favor of one that has some functional dependencies—that is, one that is not in
third normal form or higher. Think of normalization as typically being good for updating but potentially bad for
querying. Start with a normalized design and then look at all the demands that will be placed on the system.
NOTE
There really isn't a binary concept of being "normalized" or "not normalized." There are only degrees of normalization.
It is common to refer to a database that is at least in third normal form as normalized and to refer to a database at a
lower level of normalization as unnormalized or denormalized. To keep the discussion simple, I'll use the terms in
that way, as imprecise as that might be. Also note that the terms unnormalized and denormalized have slightly
This document is created with the unregistered version of CHM2PDF Pilot

different meanings. An unnormalized database is one that has never been normalized. A denormalized database is one
that was normalized at some point, but for specific performance-related reasons the design was backed down from
the normalized version. My apologies to those who make a living doing entity-relationship diagrams and are horrified
by the loose use of these terms.

If you understand the data elements you need to record and you understand data modeling, producing a normalized
design is not difficult. But it might take some time to learn about the way a business operates. If you already
understand the underlying processes to be modeled and know how to do data modeling, the mechanics of producing
a normalized database design are quite straightforward.

Once you produce a normalized design, which you can also think of as the logical design, you must decide if you
can implement the design nearly "as is" or if you need to modify it to fit your performance characteristics. A lot of
people have trouble with this. Rather than try to articulate specific performance characteristics, they generalize and
strive for "as fast as possible" or "as many users as we can handle." Although goals can be difficult to articulate
precisely, you should at least set relative goals. You should understand the tradeoffs between update and query
performance, for example. If a salesperson must call up all of a customer's records while the customer is waiting on
the phone, that action should be completed within a few seconds. Or if you want to run a bunch of batch processes
and reports for your manufacturing operation each night and you have a window of four hours in which to do it, you
have a pretty clear objective that must be met.

Evaluate Your Critical Transactions

One thing that you should do immediately is look at your critical transactions—that is, transactions whose
performance will make or break the system. (In this context, I use the term transaction loosely, to mean any
operation on the database.) Which tables and joins will be required for your critical transactions? Will data access be
straightforward or complicated?

For example, if it is imperative that a given query have less than a 2-second response time but your normalized design
would require a seven-way join, you should look at what denormalizing would cost. If tables are properly indexed,
the query is well qualified, the search parameters are quite selective, and not a lot of data needs to be returned, the
quick response might be possible. But you should note any seven-way joins and consider other alternatives. You
should probably look for alternatives any time you get beyond a four-way join.

In our example, you might decide to carry a little redundant information in a couple of tables to make it just a
three-way join. You'll incur some extra overhead to correctly update the redundant data in multiple places, but if
update activity is infrequent or less important and the query performance is essential, altering your design is probably
worth the cost. Or you might decide that rather than compute a customer's balance by retrieving a large amount of
data, you can simply maintain summary values. You can use triggers to update the values incrementally when a
customer's records change. For example, you can take the old value and add to or average it but not compute the
whole thing from scratch each time. When you need the customer balance, it is available, already computed. You
incur extra update overhead for the trigger to keep the value up-to-date, and you need a small amount of additional
storage.

Proper indexes are extremely important for getting the query performance you need. But you must face
query-vs.-update tradeoffs similar to those described earlier because indexes speed up retrieval but slow down
updating. Chapter 9 explains the extra work required when your updates require index maintenance. You might want
to lay out your critical transactions and look for the likely problems early on. If you can keep joins on critical
transactions to four tables or less and make them simple equijoins on indexed columns, you'll be in good shape.

None of these considerations is new, nor are they specific to SQL Server. Back in the mainframe days, there was a
technique known as "completing a CRUD chart." CRUD stands for Create-Retrieve-Update-Delete. In SQL, this
would translate as ISUD—Insert-Select-Update-Delete. Conceptually, CRUD is pretty simple. You draw a matrix
with critical transactions on the vertical axis and tables with their fields on the horizontal axis. The matrix gets very big
very quickly, so creating it in Microsoft Excel or in your favorite spreadsheet program can be helpful. For each
This document is created with the unregistered version of CHM2PDF Pilot

transaction, you note which fields must be accessed and how they will be accessed, and you note the access as any
combination of I, S, U, or D, as appropriate. You make the granularity at the field level so you can gain insight into
what information you want in each table. This is the information you need if you decide to carry some fields
redundantly in other tables to reduce the number of joins required. Of course, some transactions require many tables
to be accessed, so be sure to note whether the tables are accessed sequentially or via a join. You should also indicate
the frequency and time of day that a transaction runs, its expected performance, and how critical it is that the
transaction meet the performance metric.

How far you carry this exercise is up to you. You should at least go far enough to see where the potential hot spots
are for your critical transactions. Some people try to think of every transaction in the system, but that's nearly
impossible. And what's more, it doesn't matter: only a few critical transactions need special care so they don't lead to
problems. For example, if you have to do frequent select operations simultaneously on the tables that are updated
most often, you might be concerned about locking conflicts. You need to consider what transaction isolation level to
use and whether your query can live with Read Uncommitted and not conflict with the update activity. In addition, you
shouldn't worry much about such things as noncritical reports that run only during off-hours.

If you're doing complex joins or expensive aggregate functions—SUM, AVG, and so on—for common or critical
queries, you should explore techniques such as the following and you should understand the tradeoffs between query
performance improvement and the cost to your update processes:
 Add logically redundant columns to reduce the number of tables to be joined.
 Use triggers to maintain aggregate summary data, such as customer balances, the highest value, and so forth.
Such aggregates can usually be incrementally computed quickly. The update performance impact can be
slight, but the query performance improvement can be dramatic.
 Define indexes on computed columns to maintain calculated data within a single table.
 Create indexed views to automatically maintain aggregated data or data from multiple tables that might need
to be joined together. Just as with triggers, the update performance impact can be minimal, but the query
performance improvement can be dramatic.

Keep Table Row Lengths and Keys Compact

When you create tables, you must understand the tradeoffs of using variable-length columns. (See Chapter 6.) As a
general rule, data with substantial variance in the actual storage length is appropriate for variable-length columns. Also
remember that the more compact the row length, the more rows will fit on a given page. Hence, a single I/O operation
with compact rows is more efficient than an I/O operation with longer row lengths—it returns more rows and the data
cache allows more rows to fit into a given amount of memory. In addition, if a variable-length column starts out with a
small size and an UPDATE causes it to grow, the new size can end up causing the page to split, which can lead to
extra overhead.

As with tables, when you create keys you should try to make the primary key field compact because it frequently
occurs as a foreign key in other tables. If no naturally compact primary key exists, you might consider using an
identity or uniqueidentifier column as a surrogate. And recall that if the primary key is a composite of multiple
columns, the columns are indexed in the order that they are declared to the key. The order of the columns in the key
can greatly affect how selective, and hence how useful, the index is.

Your clustered key should also be as compact as possible. If your clustered key is also your primary key (which is
the default when you declare a PRIMARY KEY constraint), you might already have made it compact for the reason
mentioned above. There are additional considerations for your clustered key because SQL Server automatically
keeps the clustered key in all nonclustered indexes, along with the corresponding nonclustered key. For example, if
your clustered index is on zipcode and you have a nonclustered index on employee_id, every row in the nonclustered
index stores the corresponding zipcode value along with the employee_id value. I discussed the structure of indexes
in Chapters 3 and 8, and I'll look at the topic again later in this chapter when I discuss how to choose the best
indexes.

Occasionally, a table will have some columns that are infrequently used or modified and some that are very hot. In
This document is created with the unregistered version of CHM2PDF Pilot

such cases, it can make sense to break the single table into two tables; you can join them back together later. This is
kind of the reverse of denormalization as you commonly think of it. In this case, you do not carry redundant
information to reduce the number of tables; instead, you increase the number of tables to more than are logically called
for to put the hot columns into a separate, narrower table. With the more compact row length, you get more rows per
page and potentially a higher cache-hit ratio. As with any deviation from the normalized model, however, you should
do this only if you have good reason. After you complete a CRUD chart analysis, you might see that while your
customer table is frequently accessed, 99 percent of the time this access occurs just to find out a customer's credit
balance. You might decide to maintain this balance via a trigger rather than by recomputing it each time the query
occurs. Information such as customer addresses, phone numbers, e-mail addresses, and so on are large fields that
make the table have a wide row length. But not all that information is needed for critical transactions—only the
customer balance is needed. In this case, splitting the table into two might result in the difference between fitting, say,
150 rows on a page instead of only 2 or 3 rows. A narrower table means a greater likelihood that the customer
balance can be read from cache rather than by requiring physical I/O.
This document is created with the unregistered version of CHM2PDF Pilot

34

Planning for Peak Usage

People often ask questions such as, "Can SQL Server handle our system? We do a million transactions a day." To
answer this question, you have to know exactly what transactions they have in mind. You also need to know the
system's peak usage. If a million transactions a day are nicely spread out over 24 hours, that's less than 12
transactions per second. In general, a 12-TPS (transactions-per-second) system would be adequate. But if 90
percent of the million transactions come between 2 p.m. and 3 p.m., it's a very different situation. You'll have rates of
about 275 TPS during that hour and probably peaks of more than 350 TPS. You can use benchmarks, such as
Debit-Credit, in which SQL Server performs over 1500 TPS, but all transactions are different. It is meaningless to
refer to transactions per second in SQL Server or any other system without also talking about the types of
transactions. This is precisely the reason that standardized tests are available from the Transaction Processing Council
for comparing database performance.

Regardless of the specific transactions, though, you must design and build your system to handle peak usage. In the
case above, the important consideration is peak usage, which determines whether you should target your system to a
volume of 350 TPS for unevenly distributed usage or to only 12 TPS with usage spread out evenly. Most systems
experience peaks, and daily usage is not so nicely spread out.
This document is created with the unregistered version of CHM2PDF Pilot

34

Perceived Response Time for

Interactive Systems
Systems are often built and measured without the appropriate performance goals in mind. When measuring query
performance, for example, most designers tend to measure the time that it takes for the query to complete. By default,
this is how SQL Server decides to cost query performance. But this might not be the way your users perceive system
performance. To users, performance is often measured by the amount of time that passes between pressing the Enter
key and getting some data. As a program designer, you can use this to your advantage. For example, you can make
your application begin displaying results as soon as the first few rows are returned; if many rows will appear in the
result set, you don't have to wait until they are all processed. You can use such approaches to dramatically improve
the user's perception of the system's responsiveness. Even though the time required to get the last row might be about
the same with both approaches, the time it takes to get the first row can be different—and the perceived difference
can translate into the success or failure of the project.

By default, SQL Server optimizes a query based on the total estimated cost to process the query to completion.
Recall from Chapter 3 that if a significant percentage of the rows in a table must be retrieved, it is better to scan the
entire table than to use a nonclustered index to drive the retrieval. A clustered index, of course, would be ideal
because the data would be physically ordered already. The discussion here pertains only to the performance tradeoff
of scan-and-sort vs. using a nonclustered index.

Retrieving a page using the nonclustered index requires traversing the B-tree to get the address of a data page or the
clustering key and then retrieving that page (by using the RID to directly access it or traversing the clustered index to
find the data page) and then traversing the nonclustered B-tree again to get the location information for the next data
page and retrieving it?and so on. Many data pages are read many times each, so the total number of page accesses
can be more than the total number of pages in the table. If your data and the corresponding nonclustered index are not
highly selective, SQL Server usually decides not to use that nonclustered index. That is, if the index is not expected to
eliminate more than about 90 percent of the pages from consideration, it is typically more efficient to simply scan the
table than to do all the extra I/O of reading B-trees for both the nonclustered and clustered indexes as well as the data
pages. And by following the index, each data page must frequently be accessed multiple times (once for every row
pointed to by the index). Subsequent reads are likely to be from cache, not from physical I/O, but this is still much
more costly than simply reading the page once for all the rows it contains (as happens in a scan).

Scanning the table is the strategy SQL Server chooses in many cases, even if a nonclustered index is available that
could be used to drive the query, and even if it would eliminate a sort to return the rows based on an ORDER BY
clause. A scan strategy can be much less costly in terms of total I/O and time. However, the choice of a full scan is
based on SQL Server's estimate of how long it would take the query to complete in its entirety, not how long it would
take for the first row to be returned. If an index exists with a key that matches the ORDER BY clause of the query
and the index is used to drive the query execution, there is no need to sort the data to match the ORDER BY clause
(because it's already ordered that way). The first row is returned faster by SQL Server chasing the index even though
the last row returned might take much longer than if the table were simply scanned and the chosen rows sorted.

In more concrete terms, let's say that a query that returns many rows takes 1 minute to complete using the
scan-and-sort strategy and 2 minutes using a nonclustered index. With the scan-and-sort strategy, the user doesn't see
the first row until all the processing is almost done—for this example, about 1 minute. But with the index strategy, the
user sees the first row within a subsecond—the time it takes to do, say, five I/O operations (read two levels of the
nonclustered index, two levels of the clustered index, and then the data page). Scan-and-sort is faster in total time, but
the nonclustered index is faster in returning the first row.
This document is created with the unregistered version of CHM2PDF Pilot

SQL Server provides a query hint called FAST that lets SQL Server know that having the first n rows returned
quickly is more important than the total time, which would be the normal way query plans are costed. Later in this
chapter, we'll look at some techniques for speeding up slow queries and discuss when it's appropriate to use the
query hint. For now, you should understand the issues of response time (the time needed to get the first row) vs.
throughput (the time needed to get all rows) when you think about your performance goals. Typically, highly
interactive systems should be designed for best response time and batch-oriented systems should be designed for best
throughput.
This document is created with the unregistered version of CHM2PDF Pilot

34

Prototyping, Benchmarking, and

Testing
As you make changes to your application design or hardware configuration, you should measure the effects of these
changes. A simple benchmark test to measure differences is a tremendous asset. The benchmark system should
correlate well with the expected performance of the real system, but it should be relatively easy to run. If you have a
development "acceptance test suite" that you run before checking in any significant changes, you should add the
benchmark to that test suite.
TIP
You should measure performance with at least a proxy test; otherwise, you're setting yourself up for failure. Optimism
without data to back it up is usually misguided.

Your benchmark doesn't have to be sophisticated initially. You can first create your database and populate it with a
nontrivial amount of data—thousands of rows at a minimum. The data can be randomly generated, although the more
representative you can make the data the better. Ideally, you should use data from the existing system, if there is one.
For example, if a particular part represents 80 percent of your orders, you shouldn't make all your test data randomly
dispersed. Any differences in the selectivity of indexes between your real data and the test data will probably cause
significant differences in the execution plans you choose. You should also be sure that you have data in related tables
if you use Foreign Key constraints. As I explained earlier in this book, the enforcement of Foreign Key constraints
requires that those related tables (either referenced or referencing) be accessed if you're modifying data in a column
that is participating in the constraint. So the execution plan is sometimes considerably more complicated due to the
constraints than might be apparent, and a plethora of constraints can result in a system that has no simple operations.

As a rule of thumb, you should start with at least enough data so the difference between selecting a single row based
on its primary key by using an index is dramatically faster than selecting such a row using a table scan. This assumes
that the table in production will be large enough to reflect that difference. Remember that the system will perform
much differently depending on whether I/O operations are physical or from the cache. So don't base your conclusions
on a system that is getting high cache-hit ratios unless you have enough data to be confident that this behavior also will
be true for your production system. In addition, keep in mind that running the same test multiple times might yield
increasingly short response times. If you're testing on a dedicated machine, no other processes will be using SQL
Server's memory, and the data you read in from the disk the first time will already be in cache for subsequent tests.
TIP
If you want to run your tests repeatedly under the same conditions, use the command DBCC
DROPCLEANBUFFERS after each test run to remove all data from memory and DBCC FREEPROCCACHE to
remove all query plans from memory.

Early in the development process, you should identify areas of lock contention between transactions and any specific
queries or transactions that take a long time to run. SQL Profiler, discussed in Chapter 17, can be a wonderful tool
for tracking down your long-running queries. And if table scans will be a drain on the production system, you should
have enough data early on so that the drain is apparent when you scan. If you can run with several thousand rows of
data without lock contention problems and with good response time on queries, you're in a good position to proceed
with a successful development cycle. Of course, you must continue to monitor and make adjustments as you ramp up
to the actual system and add more realistic amounts of data and simultaneous users. And, of course, your system test
should take place on an ongoing basis before you deploy your application. It's not a one-time thing that you do the
night before you go live with a new system.

Obviously, if your smallish prototype is exhibiting lock contention problems or the queries do not perform well within
This document is created with the unregistered version of CHM2PDF Pilot

your desired goals, it's unlikely that your real system will perform as desired. Run the stored procedures that constitute
your critical transactions. You can use a simple tool such as Osql.exe to dispatch them. First run each query or
transaction alone; time it in isolation and check the execution plans using one of the SHOWPLAN options. Then run
multiple sessions to simulate multiple users, either by manually firing off multiple OSQL commands or by using a
third-party benchmarking or load stress tool. The SQL Server 2000 Resource Kit might include some tools for
creating test simulations, but as of this writing the content of the Resource Kit has not been finalized. A bit later, we'll
look at how to analyze and improve a slow-running query.
TIP
Before you roll out your production system, you should be able to conduct system tests with the same volumes of
data and usage that the real system will have when it goes live. Crossing your fingers and hoping is not good enough.
One handy tool for generating sample data, which is from the previous version of the Microsoft BackOffice Resource
Kit, is included on this book's CD. The tool, called Filltabl.exe, allows you to add any number of rows to any table,
as long as the table already exists, and the user running the utility has insert permission on the table. If you run this
executable from a command prompt, it will give you a usage message like this:
usage: filltabl -Tdbtable -Ncount [-Uusername] [-Ppassword]
[-Sservername] [-Rrandomseed]
note: there is no space between the option and the parameter
example: filltabl -Tpubs..authors -N20 -Usa -SMyServer

Even though this utility was written for an older version of SQL Server, it will work with a named instance. Just use
ServerName\InstanceName for the name of the server.

Also, based on your CRUD chart analysis (or on any other analysis of critical transactions), you should identify tasks
that will run at the same time as your critical transactions. Add these tasks to your testing routine to determine whether
they will lead to contention when they run simultaneously with your critical transactions. For example, suppose
proc_take_orders is your critical transaction. When it runs, some reports and customer status inquiries will also run.
You should run some mixture of these types of processes when you analyze proc_take_orders. This will help you
identify potential lock contention issues or other resource issues, such as high CPU usage or low cache.

You might also want to use benchmarking tools to launch and coordinate multiple client tasks simultaneously to time
their work for throughput and response time. One such suite of benchmarking tools is available from Bluecurve, Inc.
(now a subsidiary of Red Hat, Inc.), at www.redhat.com/services/bluecurve/bluecurve.html.

Development Methodologies
How much you spec, how you spec, when you start coding, and the role of prototyping are all matters on which
there is no consensus; there is no one right answer. What's right for one team or project might be wrong for another.
You must remember that the point of development is to ship products or deploy applications. Your ultimate purpose
is not to produce specs and design documents.

The spec exists to clearly articulate and document how a module or system should work. It ensures that the
developer thinks through the approach before writing code. It is also vitally important to others who come in later and
are new to the system. While writing the design document, the developer might have to write some quick prototype
code to help think through an issue. Or the developer should at least write some pseudocode and include it as part of
the document.

Any development task that will take more than a couple of days to implement deserves a simple design document.
Ideally, such a document should be at most 15 pages, and often about 3 succinct pages are ideal for even a complex
component. The best design documents can be read in one sitting, and the reader should come away understanding
clearly how the system will be built. Other developers should review the document before coding begins. This is best
done in a positive, informal atmosphere in which others can contribute ideas. And, of course, in a healthy environment,
developers are continually bouncing ideas off their peers.
This document is created with the unregistered version of CHM2PDF Pilot

The document should assume that the reader already knows why a module or component is needed. The document
provides the how. Too many specs have 40 pages describing the market need for something or why a potential
product would be really cool, and then they have 1 page that says, in essence, "We'll figure it out when we code it
up."

No one can write a perfect spec up front. Prototyping the spec in an iterative fashion works far better. Areas that are
clear don't need a prototype; use prototypes for areas fraught with risk, for your critical transactions, and for critical
areas of the system in which multiple approaches are possible or reasonable. For the tricky stuff that you can't
describe or predict a best performance, prototypes provide enormous benefits.

A useful prototype can be "quick and dirty." If you don't worry about all the failure cases and every conceivable state
in which something can exist, useful prototype code can often be produced in 5 percent of the time that it takes to
create production-caliber code (in which you must worry about those things). However, the production code might be
10 times better because you have recognized and corrected deficiencies early on. You'll either junk the prototype
code or use it for the skeleton of the real system. Prototyping lets you learn and prove or disprove ideas, and then you
can update the spec based on what you learn. And if you're selling your ideas to management or customers, your
prototype can be useful to demonstrate proof of your concept.

As I mentioned before, every nontrivial development task deserves a brief design document. You need enough
documentation to lay out the framework and system architecture and to detail how pieces fit together. But at the
detailed levels, it's better to simply comment the source code liberally while you write and modify it. External design
documentation rarely gets updated in a timely manner, so it quickly becomes useless. Explain operations when their
purpose is not obvious. Don't assume that your readers will immediately grasp all the subtleties. And when you change
something after the fact, add a comment about what you changed, when you changed it, and why. A module should
be commented well enough so that testers or technical writers can gain a good understanding of what's going on just
from reading the comments, even if they're not skilled programmers.
This document is created with the unregistered version of CHM2PDF Pilot

34

Creating Useful Indexes

Creating useful indexes is one of the most important tasks you can do to achieve good performance. Indexes can
dramatically speed up data retrieval and selection, but they are a drag on data modification because along with
changes to the data, the index entries must also be maintained and those changes must be logged. The key to creating
useful indexes is understanding the uses of the data, the types and frequencies of queries performed, and how queries
can use indexes to help SQL Server find your data quickly. A CRUD chart or similar analysis technique can be
invaluable in this effort. You might want to quickly review the difference between clustered and nonclustered indexes
because the difference is crucial in deciding what kind of index to create.

Clustered and nonclustered indexes are similar at the upper (node) levels—both are organized as B-trees. Index
rows above the leaf level contain index key values and pointers to pages the next level down. Each row keeps track
of the first key value on the page it points to. Figure 16-1 shows an abstract view of an index node for an index on a
customer's last name. The entry Johnson indicates page 1:200 (file 1, page 200), which is at the next level of the
index. Since Johnson and Jones are consecutive entries, all the entries on page 1:200 have values between Johnson
(inclusive) and Jones (exclusive).

Figure 16-1. An index node page.

The leaf, or bottom, level of the index is where clustered and nonclustered indexes differ. For both kinds of indexes,
the leaf level contains every key value in the table on which the index is built, and those keys are in sorted order. In a
clustered index, the leaf level is the data level, so of course every key value is present. This means that the data in a
table is sorted in order of the clustered index. In a nonclustered index, the leaf level is separate from the data. In
addition to the key values, the index rows contain a bookmark indicating where to find the actual data. If the table has
a clustered index, the bookmark is the clustered index key that corresponds to the nonclustered key in the row. (If the
clustered key is composite, all parts of the key are included.)

Remember that clustered indexes are guaranteed to be unique in SQL Server 2000; if you don't declare them as
unique, SQL Server adds a uniqueifier to every duplicate key to turn the index into a unique composite index. If our
index on last name is a nonclustered index and the clustered index on the table is the zip code, a leaf-level index page
might look something like Figure 16-2. The number in parentheses after the zip code is the uniqueifier and appears
only when there are duplicate zip codes.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 16-2. A leaf-level index page.

Choose the Clustered Index Carefully

Clustered indexes are extremely useful for range queries (for example, WHERE sales_quantity BETWEEN 500
and 1000) and for queries in which the data must be ordered to match the clustering key. Only one clustered index
can exist per table, since it defines the physical ordering of the data for that table. Since you can have only one
clustered index per table, you should choose it carefully based on the most critical retrieval operations. Because of the
clustered index's role in managing space within the table, nearly every table should have one. And if a table has only
one index, it should probably be clustered.

If a table is declared with a primary key (which is advisable), by default the primary key columns form the clustered
index. Again, this is because almost every table should have a clustered index, and if the table has only one index, it
should probably be clustered. But if your table has several indexes, some other index might better serve as the
clustered index. This is often true when you do single-row retrieval by primary key. A nonclustered, unique index
works nearly as well in this case and still enforces the primary key's uniqueness. So save your clustered index for
something that will benefit more from it by adding the keyword NONCLUSTERED when you declare the
PRIMARY KEY constraint.

Make Nonclustered Indexes Highly Selective

A query using an index on a large table is often dramatically faster than a query doing a table scan. But this is not
always true, and table scans are not all inherently evil. Nonclustered index retrieval means reading B-tree entries to
determine the data page that is pointed to and then retrieving the page, going back to the B-tree, retrieving another
data page, and so on until many data pages are read over and over. (Subsequent retrievals can be from cache.) With
a table scan, the pages are read only once. If the index does not disqualify a large percentage of the rows, it is
cheaper to simply scan the data pages, reading every page exactly once.

The query optimizer greatly favors clustered indexes over nonclustered indexes, because in scanning a clustered index
the system is already scanning the data pages. Once it is at the leaf of the index, the system has gotten the data as
well. So there is no need to read the B-tree, read the data page, and so on. This is why nonclustered indexes must be
able to eliminate a large percentage of rows to be useful (that is, highly selective), whereas clustered indexes are useful
even with less selectivity.

Indexing on columns used in the WHERE clause of frequent or critical queries is often a big win, but this usually
depends on how selective the index is likely to be. For example, if a query has the clause WHERE last_name =
'Stankowski', an index on last_name is likely to be very useful; it can probably eliminate 99.9 percent of the rows
from consideration. On the other hand, a nonclustered index will probably not be useful on a clause of WHERE sex =
This document is created with the unregistered version of CHM2PDF Pilot

'M' because it eliminates only about half of the rows from consideration; the repeated steps needed to read the B-tree
entries just to read the data require far more I/O operations than simply making one single scan through all the data.
So nonclustered indexes are typically not useful on columns that do not have a wide dispersion of values.

Think of selectivity as the percentage of qualifying rows in the table (qualifying rows/total rows). If the ratio of
qualifying rows to total rows is low, the index is highly selective and is most useful. If the index is used, it can eliminate
most of the rows in the table from consideration and greatly reduce the work that must be performed. If the ratio of
qualifying rows to total rows is high, the index has poor selectivity and will not be useful. A nonclustered index is most
useful when the ratio is around 5 percent or less—that is, if the index can eliminate 95 percent of the rows from
consideration. If the index has less than 5 percent selectivity, it probably will not be used; either a different index will
be chosen or the table will be scanned. Recall that each index has a histogram of sampled data values for the index
key, which the query optimizer uses to estimate whether the index is selective enough to be useful to the query.

Tailor Indexes to Critical Transactions

Indexes speed data retrieval at the cost of additional work for data modification. To determine a reasonable number
of indexes, you must consider the frequency of updates vs. retrievals and the relative importance of the competing
types of work. If your system is almost purely a decision-support system (DSS) with little update activity, it makes
sense to have as many indexes as will be useful to the queries being issued. A DSS might reasonably have a dozen or
more indexes on a single table. If you have a predominantly online transaction processing (OLTP) application, you
need relatively few indexes on a table—probably just a couple carefully chosen ones.

Look for opportunities to achieve index coverage in queries, but don't get carried away. An index "covers" the query
if it has all the data values needed as part of the index key. For example, if you have a query such as SELECT
emp_name, emp_sex from employee WHERE emp_name LIKE 'Sm%' and you have a nonclustered index on
emp_name, it might make sense to append the emp_sex column to the index key as well. Then the index will still be
useful for the selection, but it will already have the value for emp_sex. The query optimizer won't need to read the
data page for the row to get the emp_sex value; the query optimizer is smart enough to simply get the value from the
B-tree key. The emp_sex column is probably a char(1), so the column doesn't add greatly to the key length, and this
is good.

Every nonclustered index is a covering index if all you are interested in is the key column of the index. For example, if
you have a nonclustered index on first name, it covers all these queries:
 Select all the first names that begin with K.
 Find the first name that occurs most often.
 Determine whether the table contains the name Melissa.

In addition, if the table also has a clustered index, every nonclustered index includes the clustering key. So it can also
cover any queries that need the clustered key value in addition to the nonclustered key. For example, if our
nonclustered index is on the first name and the table has a clustered index on the last name, the following queries can
all be satisfied by accessing only leaf pages of the B-tree:
 Select Tibor's last name.
 Determine whether any duplicate first and last name combinations exist.
 Find the most common first name for people with the last name Wong.

You can go too far and add all types of fields to the index. The net effect is that the index becomes a virtual copy of
the table, just organized differently. Far fewer index entries fit on a page, I/O increases, cache efficiency is reduced,
and much more disk space is required. The covered queries technique can improve performance in some cases, but
you should use it with discretion.

A unique index (whether nonclustered or clustered) offers the greatest selectivity (that is, only one row can match), so
it is most useful for queries that are intended to return exactly one row. Nonclustered indexes are great for single-row
accesses via the PRIMARY KEY or UNIQUE constraint values in the WHERE clause.
This document is created with the unregistered version of CHM2PDF Pilot

Indexes are also important for data modifications, not just for queries. They can speed data retrieval for selecting
rows, and they can speed data retrieval needed to find the rows that must be modified. In fact, if no useful index for
such operations exists, the only alternative is for SQL Server to scan the table to look for qualifying rows. Update or
delete operations on only one row are common; you should do these operations using the primary key (or other
UNIQUE constraint index) values to be assured that there is a useful index to that row and no others.

A need to update indexed columns can affect the update strategy chosen. For example, to update a column that is
part of the key of the clustered index on a table, you must process the update as a delete followed by an insert rather
than as an update-in-place. When you decide which columns to index, especially which columns to make part of the
clustered index, consider the effects the index will have on the update method used. (Review the discussion of updates
in Chapter 9.)

SQL Server 2000 provides a new function that can give us a workaround if you need to index a large character field.
The CHECKSUM function computes a checksum on a row or a column, and its value is always a 4-byte integer.
You can create a computed column to be the checksum of your large character field and then build an index on that
computed column. The values returned by CHECKSUM are not guaranteed to be absolutely unique, but there will be
few duplicates. Since there is the possibility of two character string values having the same value for the checksum,
your queries will need to include the full string that you're looking for.

Here's an example. Suppose we're expecting the titles table in the pubs database to grow to over a million titles by
the end of the year. We want to be able to do quick lookups on book names, but since the title field is 80 characters
wide, we know that an index on title is not ideal. (In fact, the titles table does have an index on title, but since it's not
really such a good idea, we'll drop it.) So we'll create a computed column using CHECKSUM and index that:
DROP INDEX titles.titleind
GO
ALTER TABLE titles
ADD hash_title AS CHECKSUM(title)
GO
CREATE INDEX hash_index ON titles(hash_title)
GO

First, let's try just searching for a particular title:

SELECT * FROM titles
WHERE title = 'Cooking with Computers: Surreptitious Balance Sheets'

If you look at the query plan for this SELECT, you'll see that a clustered index scan is done, which means the whole
table must be searched. Instead, let's also query on the checksum value in the hash_title column:
SELECT * FROM titles
WHERE title = 'Cooking with Computers: Surreptitious Balance Sheets'
AND hash_title =
CHECKSUM('Cooking with Computers: Surreptitious Balance Sheets')

The query plan will now show that SQL Server can do an index seek on the computed column hash_title. We'll
look at query plans in detail later in this chapter.

Pay Attention to Column Order

At the risk of stating the obvious, an index can be useful to a query only if the criteria of the query match the columns
that are leftmost in the index key. For example, if an index has a composite key of last_name,first_name, that index
is useful for a query such as WHERE last_name = 'Smith' or WHERE last_name = 'Smith' AND first_name =
'John'. But it is not useful for a query such as WHERE first_name = 'John'. Think of using the index like a phone
This document is created with the unregistered version of CHM2PDF Pilot

book. You use a phone book as an index on last name to find the corresponding phone number. But the standard
phone book is useless if you know only a person's first name because the first name might be located on any page.

You should put the most selective columns leftmost in the key of nonclustered indexes. For example, an index on
emp_name,emp_sex is useful for a clause such as WHERE emp_name = 'Smith' AND emp_sex = 'M'. But if the
index is defined as emp_sex,emp_name, it isn't useful for most retrievals. The leftmost key, emp_sex, cannot rule out
enough rows to make the index useful. Be especially aware of this when it comes to indexes that are built to enforce a
PRIMARY KEY or UNIQUE constraint defined on multiple columns. The index is built in the order that the columns
are defined for the constraint. So you should adjust the order of the columns in the constraint to make the index most
useful to queries; doing so will not affect its role in enforcing uniqueness.

Index Columns Used in Joins

Index columns are frequently used to join tables. When you create a PRIMARY KEY or UNIQUE constraint, an
index is automatically created for you. But no index is automatically created for the referencing columns in a
FOREIGN KEY constraint. Such columns are frequently used to join tables, so they are almost always among the
most likely ones on which to create an index. If your primary key and foreign key columns are not naturally compact,
consider creating a surrogate key using an identity column (or a similar technique). As with row length for tables, if
you can keep your index keys compact, you can fit many more keys on a given page, which results in less physical
I/O and better cache efficiency. And if you can join tables based on integer values such as an identity, you avoid
having to do relatively expensive character-by-character comparisons. Ideally, columns used to join tables are integer
columns—fast and compact.

I talked about density in the statistics discussion in Chapter 15. When determining the density of a SARG, the query
optimizer can look up a specific value in the statistics histogram to get an estimate of the number of occurrences.
However, when it looks at whether an index is useful for a join operation, there is no specific value that it can look up.
The query optimizer needs to know how many rows in one table are likely to match the join columns value from the
other table. In other words, if two tables are related in a one-to-many relationship, how many is "many"? We use the
term join density to mean the average number of rows in one table that match a row in the table it is being joined to.
You can also think of join density as the average number of duplicates for an index key. A column with a unique index
has the lowest possible join density (there can be no duplicates) and is therefore extremely selective for the join. If a
column being joined has a large number of duplicates, it has a high density and is not very selective for joins.

As you learned in Chapter 15, joins are frequently processed as nested loops. For example, if while joining the
orders table with order_items the system starts with the orders table (the outer table) and then for each qualifying
order row, the inner table is searched for corresponding rows. Think of the join being processed as, "Given a specific
row in the outer table, go find all corresponding rows in the inner table." If you think of joins in this way, you'll realize
that it's important to have a useful index on the inner table, which is the one being searched for a specific value. For
the most common type of join, an equijoin that looks for equal values in columns of two tables, the query optimizer
automatically decides which is the inner table and which is the outer table of a join. The table order that you specify
for the join doesn't matter in the equijoins case. However, the order for outer joins must match the semantics of the
query, so the resulting order is dependent on the order specified.

Create or Drop Indexes as Needed

If you create indexes but find that they aren't used, you should drop them. Unused indexes slow data modification
without helping retrieval. You can determine whether indexes are used by watching the plans produced via the
SHOWPLAN options. However, this isn't easy if you're analyzing a large system with many tables and indexes. There
might be thousands of queries that can be run and no way to run and analyze the SHOWPLAN output for all of them.
An alternative is to use the Index Tuning Wizard to generate a report of current usage patterns. The wizard is
designed to determine which new indexes to build, but you can use it simply as a reporting tool to find out what is
happening in your current system. We'll look at the wizard in the next section.
This document is created with the unregistered version of CHM2PDF Pilot

Some batch-oriented processes that are query intensive can benefit from certain indexes. Such processes as complex
reports or end-of-quarter financial closings often run infrequently. If this is the case, remember that creating and
dropping indexes is simple. Consider a strategy of creating certain indexes in advance of your batch processes and
then dropping them when those batch processes are done. In this way, the batch processes benefit from the indexes
but do not add overhead to your OLTP usage.

The Index Tuning Wizard

You've seen that the query optimizer can come up with a reasonably effective query plan even in the absence of
well-planned indexes on your tables. However, this does not mean that a well-tuned database doesn't need good
indexes. The query plans that don't rely on indexes frequently consume additional system memory, and other
applications might suffer. In addition, having appropriate indexes in place can help solve many blocking problems.

Designing the best possible indexes for the tables in your database is a complex task; it not only requires a thorough
knowledge of how SQL Server uses indexes and how the query optimizer makes its decisions, but it requires that you
be intimately familiar with how your data will actually be used. The SQL Server 2000 Index Tuning Wizard is a
powerful tool for helping you design the best possible indexes.

You can access the Index Tuning Wizard from the Wizards list in SQL Server Enterprise Manager or from the Tools
menu in SQL Profiler. You can also access the wizard from the Query menu in SQL Query Analyzer. The wizard
tunes a single database at a time, and it bases its recommendations on a workload file that you provide. The workload
file can be a file of captured trace events that contains at least the RPC and SQL Batch starting or completed events,
as well as the text of the RPCs and batches. It can also be a file of SQL statements. If you use SQL Query Analyzer
to invoke the wizard, you also have the option of tuning just the current queries in your SQL Query Analyzer window.
If you use SQL Profiler to create the workload file, you can capture all SQL statement submitted by all users over a
period of time. The wizard can then look at the data access patterns for all users, for all tables, and make
recommendations with everyone in mind.

The book's companion CD includes a whitepaper describing the Index Tuning Wizard (indextuning.doc). You can
check the Microsoft web site periodically to see if any new whitepapers have been posted. The best place to look is
www.microsoft.com/sql/index.htm#W. The CD includes a set of lab exercises in a file called Analysis Labs.doc that
allow new users to get a feel for the graphical tools in SQL Server 2000. This document includes a lab exercise for
exploring the Index Tuning Wizard. To go along with these lab exercises, the CD also contains a sample database
called PierrotImports, which is available as two files, PierrotImports.mdf and PierrotImports_log.ldf. You can
use the procedure sp_attach_db to add the PierrotImports database to your system. An example of using this
command is in the file Attach_Pierrot.sql. Finally, the CD includes a sample workload file (PierrotQueries.sql)
composed of SQL statements that access the PierrotImports database.

When the wizard gets a workload file, it tunes the first 32,767 parsable queries and produces nine reports. You can
choose to make the analysis more exhaustive or less. You can choose to have the wizard keep all existing indexes, or
you can have it come up with a totally new set of index recommendations, which might mean dropping some or all of
your existing indexes. Finally, you can choose whether or not the Index Tuning Wizard will recommend building
indexed views.

After the wizard generates its reports, which you can save to disk, you can implement the suggestions immediately,
schedule the implementation for a later time, or save the script files for the creation (and optional dropping) of indexes.
Once the scripts are saved, you can run them later or just inspect them to get ideas of possible indexes to create.

You can also use the wizard purely for analyzing your existing design. One of the reports, Index Usage Report
(Current Configuration), shows what percentage of the queries in the submitted workload make use of each of your
existing indexes. You can use this information to determine whether an index is being used at all and get rid of any
unused indexes along with their space and maintenance overhead.

The current version of the wizard cannot completely tune cross-database queries. It inspects the local tables involved
This document is created with the unregistered version of CHM2PDF Pilot

in the queries for index recommendations, but it ignores the tables in other databases. Also, if you're using Unicode
data, the wizard does not recognize Unicode constants (such as N'mystring') in your queries and cannot optimize for
them. Future versions of the wizard will address these issues and probably add new features as well.
NOTE
If you're going to use SQL Profiler to capture a workload file, you should consider tracing events at several periods
throughout the day. This can give you a more balanced picture of the kinds of queries that are actually being executed
in your database. If you define your trace using SQL Profiler procedures, you can set up a SQL Server Agent job to
start and stop the trace at predefined times, appending to the same workload file each time.

You can also run the Index Tuning Wizard from a command prompt. This capability allows you to define the tuning
activities you're interested in but schedule the activity for a later time when the system might not be so busy. Any
command prompt activity can be a job step for the SQL Server Agent to run at a predetermined schedule, or you can
use the operating system's scheduler to run the command. For details, look up the itwiz utility in SQL Server Books
Online.
This document is created with the unregistered version of CHM2PDF Pilot

34

Monitoring Query Performance

Before you think about taking some action to make a query faster, such as adding an index or denormalizing, you
should understand how a query is currently being processed. You should also get some baseline performance
measurements so you can compare behavior both before and after making your changes. SQL Server provides these
tools (SET options) for monitoring queries:
 STATISTICS IO
 STATISTICS TIME
 SHOWPLAN

You enable any of these SET options before you run a query, and they will produce additional output. Typically, you
run your query with these options set in a tool such as SQL Query Analyzer. When you're satisfied with your query,
you can cut and paste it into your application or into the script file that creates your stored procedures. If you use the
SET commands to turn these options on, they apply only to the current connection. SQL Query Analyzer provides
check boxes for turning any or all of these options on or off for all connections.

STATISTICS IO
Don't let the term statistics fool you. STATISTICS IO doesn't have anything to do with the statistics used for storing
histograms and density information in sysindexes. This option provides statistics on how much work SQL Server did
to process your query. When this option is set to ON, you get a separate line of output for each query in a batch that
accesses any data objects. (You don't get any output for statements that don't access data, such as PRINT, SELECT
the value of a variable, or call a system function.) The output from SET STATISTICS IO ON includes the values
Logical Reads, Physical Reads, Read Ahead Reads, and Scan Count.

Logical Reads
This value indicates the total number of page accesses needed to process the query. Every page is read from the data
cache, whether or not it was necessary to bring that page from disk into the cache for any given read. This value is
always at least as large and usually larger than the value for Physical Reads. The same page can be read many times
(such as when a query is driven from an index), so the count of Logical Reads for a table can be greater than the
number of pages in a table.

Physical Reads
This value indicates the number of pages that were read from disk; it is always less than or equal to the value of
Logical Reads. The value of the Buffer Cache Hit Ratio, as displayed by System Monitor, is computed from the
Logical Reads and Physical Reads values as follows:
Cache-Hit Ratio = (Logical Reads _ Physical Reads) / Logical Reads

Remember that the value for Physical Reads can vary greatly and decreases substantially with the second and
subsequent execution because the cache is loaded by the first execution. The value is also affected by other SQL
Server activity and can appear low if the page was preloaded by read ahead activity. For this reason, you probably
won't find it useful to do a lot of analysis of physical I/O on a per-query basis. When you're looking at individual
queries, the Logical Reads value is usually more interesting because the information is consistent. Physical I/O and
achieving a good cache-hit ratio is crucial, but they are more interesting at the all-server level. Pay close attention to
Logical Reads for each important query, and pay close attention to physical I/O and the cache-hit ratio for the server
This document is created with the unregistered version of CHM2PDF Pilot

as a whole.

STATISTICS IO acts on a per-table, per-query basis. You might want to see the physical_io column in
sysprocesses corresponding to the specific connection. This column shows the cumulative count of synchronous I/O
that has occurred during the spid's existence, regardless of the table. It even includes any Read Ahead Reads that
were made by that connection.

Read Ahead Reads

The Read Ahead Reads value indicates the number of pages that were read into cache using the read ahead
mechanism while the query was processed. These pages are not necessarily used by the query. If a page is ultimately
needed, a logical read is counted but a physical read is not. A high value means that the value for Physical Reads is
probably lower and the cache-hit ratio is probably higher than if a read ahead was not done. In a situation like this,
you shouldn't infer from a high cache-hit ratio that your system can't benefit from additional memory. The high ratio
might come from the read ahead mechanism bringing much of the needed data into cache. That's a good thing, but it
might be better if the data simply remains in cache from previous use. You might achieve the same or a higher
cache-hit ratio without requiring the Read Ahead Reads.

You can think of read ahead as simply an optimistic form of physical I/O. In full or partial table scans, the table's
IAMs are consulted to determine which extents belong to the object. The extents are read with a single 64 KB scatter
read, and because of the way that the IAMs are organized, they are read in disk order. If the table is spread across
multiple files in a file group, the read ahead attempts to keep at least eight of the files busy instead of sequentially
processing the files. Read Ahead Reads are asynchronously requested by the thread that is running the query; because
they are asynchronous, the scan doesn't block while waiting for them to complete. It blocks only when it actually tries
to scan a page that it thinks has been brought into cache and the read hasn't finished yet. In this way, the read ahead
neither gets too ambitious (reading too far ahead of the scan) nor too far behind.

Scan Count
The Scan Count value indicates the number of times that the corresponding table was accessed. Outer tables of a
nested loop join have a Scan Count of 1. For inner tables, the Scan Count might be the number of times "through the
loop" that the table was accessed. The number of Logical Reads is determined by the sum of the Scan Count times
the number of pages accessed on each scan. However, even for nested loop joins, the Scan Count for the inner table
might show up as 1. SQL Server might copy the needed rows from the inner table into a worktable in cache and use
this worktable to access the actual data rows. When this step is used in the plan, there is often no indication of it in the
STATISTICS IO output. You must use the output from STATISTIC TIME, as well as information about the actual
processing plan used, to determine the actual work involved in executing a query. Hash joins and merge joins usually
show the Scan Count as 1 for both tables involved in the join, but these types of joins can involve substantially more
memory. You can inspect the memusage value in sysprocesses while the query is being executed, but unlike the
physical_io value, this is not a cumulative counter and is valid only for the currently running query. Once a query
finishes, there is no way to see how much memory it used.

Troubleshooting Example Using STATISTICS IO

The following situation is based on a real problem I encountered when doing a simple SELECT * from a table with
only a few thousand rows. The amount of time this simple SELECT took seemed to be out of proportion to the size
of the table. We can create a similar situation here by making a copy of the orders table in the Northwind database
and adding a new column to it with a default value of 2000-byte character field.
SELECT *
INTO neworders
FROM orders
GO
ALTER TABLE neworders
ADD full_details CHAR(2000) NOT NULL DEFAULT 'full details'
This document is created with the unregistered version of CHM2PDF Pilot

We can gather some information about this table with the following command:
EXEC sp_spaceused neworders, true

The results will tell us that there are 830 rows in the table, 3240 KB for data, which equates to 405 data pages. That
doesn't seem completely unreasonable, so we can try selecting all the rows after enabling STATISTICS IO:
SET STATISTICS IO ON
GO
SELECT * FROM neworders

The results now show that SQL Server had to perform 1945 logical reads for only 405 data pages. There is no
possibility that a nonclustered index is being used inappropriately, because I just created this table, and there are no
indexes at all on it. So where are all the reads coming from?

Recall that if a row is increased in size so that it no longer fits on the original page, it will be moved to a new page,
and SQL Server will create a forwarding pointer to the new location from the old location. In Chapter 9, I told you
about two ways to find out how many forwarding pointers were in a table: we could use the undocumented trace flag
2509, or we could run DBCC SHOWCONTIG with the TABLERESULTS options. Here, I'll use the trace flag:
DBCC TRACEON (2509)
GO
DBCC CHECKTABLE (neworders)

RESULTS:
DBCC execution completed. If DBCC printed error messages, contact your
system administrator.
DBCC results for 'neworders'.
There are 830 rows in 405 pages for object 'neworders'.
Forwarded Record count = 770

If you play around with the numbers a bit, you might realize that 2*770 + 405 equals 1945, the number of reads we
ended up performing with the table scan. What I realized was happening was that when scanning a table, SQL Server
will read every page (there are 405 of them), and each time a forwarding pointer is encountered, SQL Server will
jump ahead to that page to read the row and then jump back to the page where the pointer was and read the original
page again. So for each forwarding pointer, there are two additional logical reads. SQL Server is doing a lot of work
here for a simple table scan.

The "fix" to this problem is to regularly check the number of forwarding pointers, especially if a table has
variable-length fields and is frequently updated. If you notice a high number of forwarding pointers, you can rebuild
the table by creating a clustered index. If for some reason I don't want to keep the clustered index, I can drop it. But
the act of creating the clustered index will reorganize the table.
SET STATISTICS IO OFF
GO
CREATE CLUSTERED INDEX orderID_cl ON neworders(orderID)
GO
DROP INDEX neworders.orderID_cl
GO
EXEC sp_spaceused neworders, true
SET STATISTICS IO ON
GO
SELECT ROWS = count(*) FROM neworders

RESULTS:
This document is created with the unregistered version of CHM2PDF Pilot

name rows reserved data index_size unused

--------------- -------- ---------- --------- ------------- ------
neworders 830 2248 KB 2216 KB 8 KB 24 KB

ROWS
-----------
830

(1 row(s) affected)

Table 'neworders'. Scan count 1, logical reads 277, physical reads 13,
read-ahead reads 278.

This output shows us that not only did the building of the clustered index reduce the number of pages in the table,
from 405 to 277, but the cost to read the table is now the same as the number of pages in the table.

STATISTICS TIME
The output of SET STATISTICS TIME ON is pretty self-explanatory. It shows the elapsed and CPU time required
to process the query. In this context, it means the time not spent waiting for resources such as locks or reads to
complete. The times are separated into two components: the time required to parse and compile the query, and the
time required to execute the query. For some of your testing, you might find it just as useful to look at the system time
with getdate before and after a query if all you need to measure is elapsed time. However, if you want to compare
elapsed vs. actual CPU time or if you're interested in how long compilation and optimization took, you must use
STATISTICS TIME.

In many cases, you'll notice that the output includes two sets of data for parse and compile time. This happens when
the plan is being added to syscacheobjects for possible reuse. The first line is the actual parse and compile before
placing the plan in cache, and the second line appears when SQL Server is retrieving the plan from cache. Subsequent
executions will still show the same two lines, but if the plan is actually being reused, the parse and compile time for
both lines will be 0.

Showplan
In SQL Server 2000, there is not just a single option for examining the execution plan for a query. You can choose to
see the plan in a text format, with or without additional performance estimates, or you can see a graphical
representation of the processing plan. In Chapter 15, I gave you a sneak preview of the graphical showplan. Now I'll
go into more detail.

The output of any of the showplan options shows some basic information about how the query will be processed.
The output will tell you, for each table in the query, whether indexes are being used or whether a table scan is
necessary. It will also tell you the order that all the different operations of the plan are to be carried out. Reading
showplan output is as much an art as it is a science, but for the most part it just takes a lot of practice to get really
comfortable with its interpretation. I hope that the examples in this section will be enough to give you a good start.

SHOWPLAN_TEXT and SHOWPLAN_ALL

The two SET options SHOWPLAN_TEXT and SHOWPLAN_ALL let you see the estimated query plan without
actually executing the query. Both options also automatically enable the SET NOEXEC option, so you don't see any
results from your query—you see only the way that SQL Server has determined is the best method for processing the
query. Turning NOEXEC ON can be a good thing while tuning a query. For example, if you have a query that takes
20 minutes to execute, you might try to create an index that will allow it to run faster. However, immediately after
creating a new index, you might just want to know whether the query optimizer will even choose to use that index. If
This document is created with the unregistered version of CHM2PDF Pilot

you were actually executing the query every time you looked at its plan, it would take you 20 minutes for every tuning
attempt. Setting NOEXEC ON along with the showplan option will allow you to see the plan without actually
executing all the statements.
WARNING
Since turning on SHOWPLAN_TEXT or SHOWPLAN_ALL implies that NOEXEC is also on, you must set the
SHOWPLAN option to OFF before you do anything else. For example, you must set SHOWPLAN_TEXT to OFF
before setting SHOWPLAN_ALL to ON.

SHOWPLAN_TEXT shows you all the steps involved in processing the query, including the type of join used, the
order of table accesses, and which index or indexes are used for each table. Any internal sorts are also shown. Here's
an example of a simple two-table join from the Northwind database:
SET SHOWPLAN_TEXT ON
GO
SELECT ProductName, Products.ProductId
FROM [Order Details] JOIN Products
ON [Order Details].ProductID = Products.ProductId
WHERE Products.UnitPrice > 10
GO

OUTPUT:
StmtText
----------------------------------------------------------------------
|--Nested Loops(Inner Join, OUTER REFERENCES:([Products].[ProductID]))
|--Clustered Index Scan
(OBJECT:([Northwind].[dbo].[Products].[PK_Products]),
WHERE:([Products].[UnitPrice]>10.00))
|--Index Seek
(OBJECT:([Northwind].[dbo].[Order Details].[ProductID]),
SEEK:([Order Details].[ProductID]=[Products].[ProductID])
ORDERED FORWARD)

The output tells us that the query plan is performing an inner join on the two tables. Inner Join is the logical operator,
as requested in the query submitted. The way that this join is actually carried out is the physical operator; in this case,
the query optimizer chose to use nested loops. The outer table in the join is the Products table, and a Clustered Index
Scan is the physical operator. In this case, there is no separate logical operator—the logical operator and the physical
operator are the same. Remember that since a clustered index contains all the table data, scanning the clustered index
is equivalent to scanning the whole table. The inner table is the Order Details table, which has a nonclustered index
on the ProductID column. The physical operator used is Index Seek. The Object reference after the Index Seek
operator shows us the full name of the index used: :([Northwind].[dbo].[Order Details].[ProductID]. In this
case, it is a bit confusing because the name of the index is the same as the name of the join column. For this reason, I
recommend not giving indexes the same names as the columns they will access.

SHOWPLAN_ALL provides this information plus estimates of the number of rows that are expected to meet the
queries' search criteria, the estimated size of the result rows, the estimated CPU time, and the total cost estimate that
was used internally when comparing this plan to other possible plans. I won't show you the output from
SHOWPLAN_ALL because it's too wide to fit nicely on a page of this book. But the information returned is the
same as the information you'll be shown with the graphical showplan, and I'll show you a few more examples of
graphical plans in this chapter.

When using showplan to tune a query, you typically add one or more indexes that you think might help speed up a
query and then you use one of these showplan options to see whether any of your new indexes were actually used. If
showplan indicates that a new index is being used, you should then actually execute the query to make sure that the
performance is improved. If an index is not going to be used or is not useful in terms of performance improvement,
and you're not adding it to maintain a Primary Key or Unique constraint, you might as well not add it. If an index is not
useful for queries, the index is just overhead in terms of space and maintenance time. After you add indexes, be sure
This document is created with the unregistered version of CHM2PDF Pilot

to monitor their effect on your updates because indexes can add overhead to data modification (inserts, deletes, and
updates).

If a change to indexing alone is not the answer, you should look at other possible solutions, such as using an index
hint or changing the general approach to the query. In Chapter 10, you saw that several different approaches can be
useful to some queries—ranging from the use of somewhat tricky SQL to the use of temporary tables and cursors. If
these approaches also fail to provide acceptable performance, you should consider changes to the database design
using the denormalization techniques discussed earlier.

Graphical Showplan
You can request a graphical representation of a query's estimated execution plan in SQL Query Analyzer either by
using the Display Estimated Execution Plan toolbar button immediately to the right of the database drop-down list or
by choosing Display Estimated Execution Plan from the Query menu. Like the SET options SHOWPLAN_TEXT
and SHOWPLAN_ALL, by default your query is not executed when you choose to display the graphical showplan
output. The graphical representation contains all the information available through SHOWPLAN_ALL, but not all of it
is visible at once. You can, however, move your cursor over any of the icons in the graphical plan to see the additional
performance estimates.

Figure 16-3 shows the graphical query plan for the same query I used in the previous section, and I've put my cursor
over the icon for the nested loop join so you can see the information box. When reading the graphical showplan, you
need to read from right to left. When you look at the plan for a JOIN, the table listed on the top is the first one
processed. For a LOOP join, the top table is the outer table, and for a HASH join, the top table is the build table.
The figure shows that the Products table is accessed first as the outer table in a Nested Loop join, and the other table
would be the inner table. You can't see the other table in the figure because the information box is covering it up. The
final step in the plan, shown on the far left, is to actually SELECT the qualifying rows and return them to the client.

Figure 16-3. A graphical showplan showing a nested loop join operation.

Table 16-1, taken from SQL Server Books Online, describes the meaning of each of the output values that you can
see in the details box for the Nested Loops icon. Some of these values, for example StmtId, NodeId, and Parent, do
not show up in the graphical representation—only in the SHOWPLAN_ALL output.

Table 16-1. Values available in the showplan output.

Column What It Contains
StmtText The text of the Transact-SQL statement.
StmtId The number of the statement in the current batch.
This document is created with the unregistered version of CHM2PDF Pilot

NodeId The ID of the node in the current query.

Parent The node ID of the parent step.
PhysicalOp The physical implementation algorithm for the node.
LogicalOp The relational algebraic operator that this node represents.
Supplemental information about the operation being
Argument performed. The contents of this column depend on the
physical operator.
A comma-separated list of values introduced by this
operator. These values can be computed expressions that
were present in the current query (for example, in the
DefinedValues SELECT list or WHERE clause) or internal values
introduced by the query processor in order to process this
query. These defined values can then be referenced
elsewhere within this query.
EstimateRows The estimated number of rows output by this operator.
EstimateIO The estimated I/O cost for this operator.
EstimateCPU The estimated CPU cost for this operator.
The estimated average row size (in bytes) of the row
AvgRowSize
being passed through this operator.
The estimated (cumulative) cost of this operation and all
TotalSubtreeCost
child operations.
A comma-separated list of columns being projected by
OutputList
the current operation.
A comma-separated list of warning messages relating to
Warnings
the current operation.
The node type. For the parent node of each query, this is
the Transact-SQL statement type (for example, SELECT,
Type
INSERT, EXECUTE, and so on). For subnodes
representing execution plans, the type is PLAN_ROW.
0 = Operator is not running in parallel.
Parallel
1 = Operator is running in parallel.
The estimated number of times this operator will be
EstimateExecutions
executed while running the current query.

Let's look at a slightly more complex query plan for this query based on a four-table join and a GROUP BY with
aggregation.
USE Northwind
SELECT e.LastName, p.ProductName, sum(d.Quantity * d.UnitPrice)
FROM [Order Details] d JOIN Orders o
ON o.OrderID = d.OrderID
JOIN Products p
ON d.ProductID = p.ProductID
JOIN Employees e
ON e.EmployeeID = o.EmployeeID
GROUP BY e.LastName, p.ProductName

You can see most of the graphical plan in Figure 16-4. Because there are four tables, the plan shows three join
operators. You need to read the plan from right to left, and you can see that the Employees and Orders tables are
joined with a Nested Loops join, and then the result of that first join is joined with the Order Details table, again using
Nested Loops. Finally, the result of the second join is joined with the Products table using hashing. Note that the
This document is created with the unregistered version of CHM2PDF Pilot

order that the joins are processed in the plan is not at all the same order used when writing the query. The query
optimizer attempts to find the best plan regardless of how the query was actually submitted. The final step is to do the
grouping and aggregation, and the graphical plan shows us that this was done using hashing.

Figure 16-4. A graphical showplan for a four-table join with GROUP BY aggregation.
WARNING
In order to generate the graphical showplan, SQL Server must make sure that both SHOWPLAN_TEXT and
SHOWPLAN_ALL are in the OFF state so that the NOEXEC option is also OFF. If you've been tuning queries
using the SHOWPLAN_TEXT option and you then decide to look at a graphical showplan, the NOEXEC option
will be left in the OFF state after you view the graphical plan. If you then try to go back to looking at
SHOWPLAN_TEXT, you'll find that it has been turned off, and any queries you run will actually be executed.

Troubleshooting with Showplan

One of the first things to look for in your showplan output is whether there are any warning messages. Warning
messages might include the string "NO STATS:()" with a list of columns. This warning message means that the query
optimizer attempted to make a decision based on the statistics for this column but that no statistics were available.
Consequently, the query optimizer had to make a guess, which might have resulted in the selection of an inefficient
query plan. Warnings can also include the string MISSING JOIN PREDICATE, which means that a join is taking
place without a join predicate to connect the tables. Accidentally dropping a join predicate might result in a query that
takes much longer to run than expected and returns a huge result set. If you see this warning, you should verify that the
absence of a join predicate is what you wanted, as in the case of a CROSS JOIN.

Another area where SHOWPLAN can be useful is for determining why an index wasn't chosen. You can look at the
estimated row counts for the various search arguments. Remember that a nonclustered index will mainly be useful if
only a very small percentage of rows in the table will qualify; if the showplan output shows hundreds of rows meeting
the condition, that can be a good indication of why a nonclustered index was not chosen. However, if the row
estimate is very different from what you know it to actually be, you can look in two places for the reason. First make
sure statistics are updated or that you haven't turned off the option to automatically update statistics on indexes.
Second, make sure your WHERE clause actually contains a SARG and that you haven't invalidated the use of the
index by applying a function or other operation to the indexed column. You saw an example of this problem in
Chapter 15, and the plan was shown in Figure 15-4.

Another thing to remember from Chapter 15 is that there is no guarantee that the query optimizer will find the
absolute fastest plan. It will search until it finds a good plan; there might actually be a faster plan, but the query
optimizer might take twice as long to find it. The query optimizer decides to stop searching when it finds a "good
enough" plan because the added cost of continued searching might offset whatever slight savings in execution time
This document is created with the unregistered version of CHM2PDF Pilot

you'd get with a faster plan.

Here's an example. We'll build a table with big rows, each of which has a char(7000), so it needs one page per row.
We'll then populate this table with 1000 rows of data and build three indexes:
CREATE TABLE emps(
empid int not null identity(1,1),
fname varchar(10) not null,
lname varchar(10) not null,
hiredate datetime not null,
salary money not null,
resume char(7000) not null)
GO

-- populate with 1000 rows, each of which will fill a page

SET NOCOUNT ON
DECLARE @i AS int
SET @i = 1
WHILE @i <= 1000
BEGIN
INSERT INTO emps VALUES('fname' + cast(@i AS varchar(5)),
'lname' + cast(@i AS varchar(5)),
getdate() - @i,
1000.00 + @i % 50,
'resume' + cast(@i AS varchar(5)))
SET @i = @i + 1
END
GO

CREATE CLUSTERED INDEX idx_ci_empid ON [dbo].[emps] ([empid])

CREATE NONCLUSTERED INDEX idx_nci_lname ON [dbo].[emps] ([lname])
CREATE NONCLUSTERED INDEX idx_nci_fname ON [dbo].[emps] ([fname])
GO

The query we'll execute will only access the fname and lname columns. As you've seen, SQL Server uses the index
intersection technique to use two nonclustered indexes to determine the qualifying rows and join the results from the
two indexes together. However, the optimizer decided not to use that plan in this case:
SELECT fname, lname
FROM emps

The plan shows a clustered index scan, which means that the entire table is examined. The number of logical reads is
about equal to the number of pages in the table (1002). However, if we apply a SARG that actually includes the
whole table, we'll find that the query optimizer will use the index intersection technique. Here's the query:
SELECT fname, lname
FROM emps
WHERE fname BETWEEN 'fname' and 'fname9999'

Figure 16-5 shows the query plan using an index seek on the nonclustered index on fname and an index scan on the
nonclustered index on lname, and then using a hash join to combine the result sets of the two index operations. If we
turn on STATISTICS IO, the logical reads are reported to be only 8 pages. It might seem as if the difference
between 8 logical reads and 1000 logical reads is a lot, but in the grand scheme of things, it really isn't all that much.
Reading 1000 pages is still a very fast operation, and the query optimizer decided not to search further. It will only
consider the index intersection case if it already is doing index seek operations, and with no SARGs in the query,
index seek is not an option.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 16-5. A graphical showplan for an index intersection when index seek operations are used.

However, if the table has more than 1000 rows, the situation changes. If we drop the emps table and re-create it
with 10,000 rows instead of 1000, the cost of doing the simple clustered index scan becomes 10,000 logical reads. In
this case, that is deemed too expensive, so the query optimizer will keep searching for a better plan. With 10,000
rows in the emps table, the index intersection technique is evaluated and found to be faster. In fact, it will take only
about 60 logical reads, which is a big improvement over 10,000.

Logical and Physical Operators

The logical and physical operators describe how a query or update was executed. The physical operators describe
the physical implementation algorithm used to process a statement—for example, scanning a clustered index or
performing a merge join. Each step in the execution of a query involves a physical operator. The logical operators
describe the relational algebraic operation used to process a statement—for example, performing an aggregation or an
inner join. Not all steps used to process a query or update involve logical operations.

Logical operators map pretty closely to what is contained in the query itself. As you've seen, you might have an inner
join for the logical operator. The physical operator is then how the query optimizer decided to process that join—for
example as a nested loop join, a hash join, or a merge join. Aggregation is a logical operator because the query
actually specifies a GROUP BY with an aggregate and the corresponding physical operator, as chosen by the query
optimizer, could be stream aggregate (if the grouping is done by sorting) or hash match (if the grouping is done by
hashing).

You might see some types of joins that you don't recognize. I've already talked about inner joins; left, right and full
outer joins; and cross joins. You might also see the following logical operators: Left Semi Join, Right Semi Join,
Left Anti Semi Join, and Right Anti Semi Join. These sound mysterious, but they're really not. Normally, in a join,
you're interested in columns from both tables. But if you use a subquery with an EXISTS or IN predicate, you're
saying that you're not interested in retrieving data from the inner table and you're using it only for lookup purposes. So
instead of needing data from both tables, you need data from only one of the tables, and this is a Semi Join. If you're
looking for data from the outer table that doesn't exist in the inner table, it becomes an Anti Semi Join.

Here's an example in the pubs database that retrieves all the names of publishers that don't have corresponding
pub_id values in the titles table—in other words, publishers who haven't published any books:
USE pubs
GO
SELECT pub_name
FROM publishers
WHERE pub_id NOT IN (SELECT pub_id FROM titles)
This document is created with the unregistered version of CHM2PDF Pilot

Here's the SHOWPLAN_TEXT output that the query generates. You can see the Left Anti Semi Join operator.
StmtText
-----------------------------------------------------------------------
|--Nested Loops(Left Anti Semi Join, WHERE:([titles].[pub_id]=NULL OR
[publishers].[pub_id]=[titles].[pub_id]))
|--Clustered Index
Scan(OBJECT:([pubs].[dbo].[publishers].[UPKCL_pubind]))
|--Clustered Index
Scan(OBJECT:([pubs].[dbo].[titles].[UPKCL_titleidind]))

Displaying the Actual Plan

All of the showplan options discussed so far show the estimated query plan. As long as the query is not actually being
executed, you can only see an estimate. As you saw in Chapter 15, conditions such as memory resources might
change before you actually run the query, so the estimated plan might not be the actual plan. If you need to see the
actual plan at the time the query is executed, you have three options:
 Choose Show Execution Plan from the Query menu in SQL Query Analyzer. Unlike the Display Estimated
Execution Plan option, this one acts like a toggle. Once you select it, it stays selected until you deselect it.
You'll see two tabs of results for every query you run until you deselect Show Execution Plan. The first tab
shows the actual query output, and the second shows the graphical showplan.
 Set STATISTICS PROFILE to ON. This option gives you the query results followed by the
STATISTICS_ALL output in the same results pane.
 Set up a trace in SQL Profiler to capture the Execution Plan events. This displays from the same information
as SHOWPLAN_TEXT in the trace output for every query that is executed. We'll look at tracing in more
detail in Chapter 17.

Other SQL Query Analyzer Tools

In SQL Server 2000, SQL Query Analyzer provides other troubleshooting information besides SHOWPLAN and
statistical output. Two additional options on the Query menu act as toggles to provide more information for every
query that you run: Show Server Trace and Show Client Statistics. Each of these options, when enabled, adds
another tab to the results window. The Server Trace output gives information similar to what you might see when you
use SQL Profiler, which I'll discuss in Chapter 17. The Client Statistics output can give you a set of very useful data
that indicates how much work the client had to do to submit this query and retrieve the results from SQL Server. It
includes information that can also be returned by other means, such as the time spent and the number of rows
affected, and it can also give additional information such as the number of bytes sent and received across the network.

Let's look at one of the Northwind queries I ran earlier and see what the client statistics look like when I execute it.
Here's the query again:
SELECT e.LastName, p.ProductName, sum(d.Quantity * d.UnitPrice)
FROM [Order Details] d JOIN Orders o
ON o.OrderID = d.OrderID
JOIN Products p
ON d.ProductID = p.ProductID
JOIN Employees e
ON e.EmployeeID = o.EmployeeID
GROUP BY e.LastName, p.ProductName

Here's the information that Client Statistics returned:

Counter Value Average
Application Profile Statistics:
This document is created with the unregistered version of CHM2PDF Pilot

Timer resolution (milliseconds) 0 0

Number of INSERT, UPDATE,
0 0
DELETE statements
Rows effected by INSERT,
0 0
UPDATE, DELETE statements
Number of SELECT statements 1 1
Rows effected by SELECT
588 588
statements
Number of user transactions 1 1
Average fetch time 0 0
Cumulative fetch time 0 0
Number of fetches 0 0
Number of open statement handles 0 0
Max number of opened statement
0 0
handles
Cumulative number of statement
0 0
handles
Network Statistics:
Number of server roundtrips 1 1
Number of TDS packets sent 1 1
Number of TDS packets received 9 9
Number of bytes sent 552 552
Number of bytes received 35,921 35,921
Time Statistics:
Cumulative client processing time 223 223
Cumulative wait time on server replies 3 3

The number in the Value column pertains only to the most recently run batch. The number in the Average column is a
cumulative average since you started collecting client statistics for this session. If you turn off the option to collect
client statistics and then turn it on again, the numbers in the Average column will be reset.

There are a few other interesting features in SQL Query Analyzer that you should know about. I don't want to go
into a lot of detail because this isn't a book about pointing and clicking your way through the tools and most of these
tools aren't directly related to tuning your queries. But the more fluent you are in your use of this tool, the more you
can use it to maximum benefit when you perform troubleshooting and tuning operations. This is not a complete list of
all the capabilities of SQL Query Analyzer. I suggest you read SQL Server Books Online thoroughly and spend
some time just exploring this tool even if you already think you know how to use it. I'll bet you'll discover something
new. For a start, take a look at these options:

Object Browser The Object Browser is a tree-based tool window for navigating through the database objects of a
selected SQL Server. Besides navigation, the Object Browser offers object scripting and procedure execution. To
limit the overhead of using the Object Browser, only one server can be active at a time. The drop-down list box at the
top of the Object Browser presents a distinct list of server names to which you can connect, and this means that you
must have a connection already open to a server before you can use the Object Browser. The Object Browser has
no explicit connect logic: it piggybacks on the existing query window connection's information.

Changing Fonts You can right-click in almost any window and choose the option to change your font. Alternatively,
you can choose Options from the Tools menu and go to the Fonts tab, where you can set the font of any type of
window.
This document is created with the unregistered version of CHM2PDF Pilot

Quick Edit From the Edit menu, you can choose to set or clear bookmarks, go to a specific line in the query
window, change text to uppercase or lowercase, indent or unindent a block of code, or comment out a whole block
of code.

Shortcut Keys You can configure the Ctrl+1 through Ctrl+0 and Alt+F1 and Ctrl+F1 keys as query shortcuts.
Choose Customize from the Tools menu to bring up a dialog box in which you can define a command or stored
procedure (either system or user defined) to be run when that key combination is entered. The command itself won't
be displayed in the query window, but the result of running the defined command will appear in the results pane of the
active query window. By default, the Alt+F1 combination executes sp_help, but you can change this.

Any text highlighted in the query window will be appended to the command that is executed as a single string. If the
command is a stored procedure, the highlighted text will then be interpreted as a single parameter to that stored
procedure. If this text is not compatible with the parameter the procedure is expecting, an error will result. To prevent
too many errors, SQL Server will check if the procedure accepts parameters; if not, no parameters will be sent,
regardless of what is highlighted in the query window.

Although the Customize dialog box indicates that it accepts only stored procedures as text, this is not completely true.
You can type in a complete SQL statement, such as SELECT getdate(). However, if you define something other
than a stored procedure as the executed text for the shortcut, any highlighted text in the query window is ignored. For
example, you might want to have the text associated with a key combination to be SELECT * FROM so that you can
type a table name into the query window, highlight it, and use the keyboard shortcut to get all the rows from that table.
(That is, you want to execute SELECT * FROM <highlighted table name>). Since the text associated with the
keystroke is not a stored procedure, your highlighted text will be ignored and you'll end up with a syntax error.

Finally, keep in mind that the command to be executed will run in the current database context. System stored
procedures will never cause any problems, but executing a local stored procedure or selecting from a table might
cause an error if you're not in the right database when you use the shortcut. Alternatively, you can fully qualify the
name of the referenced object in the shortcut definition.
TIP
There are dozens of predefined shortcuts for SQL Server 2000 SQL Query Analyzer. For a complete list, see the
file called SQL QA Shortcut Matrix.xls on the companion CD.

Customizing the Toolbar You can add additional icons to the SQL Query Analyzer toolbar by double-clicking (or
right-clicking) on any open area of the toolbar to bring up the Customize Toolbar dialog box. In the Customize
Toolbar dialog box, you can add almost any of the commands available through the menus as buttons on the toolbar.
You can also use this dialog box to change the position of existing buttons.

Object Search Object Search, available from the Tools menu, is an alternative way to locate objects besides using
the tree-based metaphor available through the Object Browser. You can specify the name of an object to search for,
the database name (or all databases), and the types of objects you're interested in. If the server is case sensitive, you
can make the search either case sensitive or case insensitive; if the server is case insensitive, the search can be case
insensitive only and the Case-Sensitive check box will be disabled. By default, the Object Search imposes a hit limit
on the search of 100 objects returned, but you can change this limit.

There can be multiple instances of the Object Search, each connected to the same or to different servers. The Object
Search is a child window like the query window. Each Object Search window has one dedicated connection. When
you open the Object Search window, it connects to the same server that the active query window is connected to.

The searching is performed using a stored procedure called sp_MSobjsearch. This procedure is undocumented, but
it exists in the master database, so you can use sp_helptext to see its definition. It is fairly well commented, at least
as far as listing what parameters it expects and the default values for those parameters.

The results of the stored procedure are displayed in the grid of the Object Search window. From there, the same
functionality is available as in the Object Browser. You can drag a row into the query editor while holding down the
This document is created with the unregistered version of CHM2PDF Pilot

secondary mouse button to initiate a script option.

Transact-SQL Debugger You can use the Transact-SQL Debugger to control and monitor the execution of stored
procedures and support traditional debugger functionality such as setting breakpoints, defining watch expressions, and
single-stepping through procedures. The debugger lets you step through nested procedure calls, and you can debug a
trigger by creating a simple procedure that performs the firing action on the trigger's table. The trigger is then treated
as a nested procedure. Note that the default behavior of the debugger is to roll back any changes that have taken
place, so even though you're executing data modification operations, the changes to your tables will not be permanent.
You can, however, change this default behavior and choose to not roll back any changes. To debug a user-defined
function (you can't debug system supplied functions because they're not written using Transact-SQL), you can write a
procedure that includes a SELECT statement containing the function call.

If you're interested in working with the debugger, I suggest that you thoroughly read the article in SQL Server Books
Online titled "Troubleshooting the Transact-SQL Debugger" and spend a bit of time getting very familiar with the tool.
To invoke the debugger, choose a particular database in the Object Browser and open the Stored Procedures folder
for that database. Right-click on a procedure name and select Debug on the context menu. If the procedure expects
parameters, you'll be prompted for them.

Here are a few issues that deserve special attention:

 Try to include the RETURN keyword in all your procedures and triggers that you might end up needing to
debug. In some cases, the debugger might not be able to correctly step back to the calling procedure if a
RETURN is not encountered.
 Because the debugger starts a connection to the server in a special mode that allows single stepping, you
cannot use the debugger to debug concurrency or blocking issues.
 To use all the features of the debugger, the SQL Server must be running under a real account and not using
Local System. The Local System special account cannot access network resources, and even if you're
running the debugger on the same machine on which the server is running, the debugger uses DCOM to
communicate between the client and the server. DCOM requires a real operating system account. The initial
release of SQL Server 2000 has an additional problem with this requirement. When the debugger is first
invoked, the code checks only the account that the default instance is running under. If that is Local System,
you'll get a warning and will be able to choose whether or not to continue. There are two problems with this.
First of all, if you're connecting to a named instance and that instance is using a valid account, you'll still get
this warning if the default instance is using Local System. Second, if the default instance is using a real account
but the instance you're connecting to is not, you won't get the warning but the debugger will not be fully
functional.

Using Query Hints

As you know, SQL Server performs locking and query optimization automatically. But because the query optimizer
is probability based, it sometimes makes wrong predictions. For example, to eliminate a deadlock, you might want to
force an update lock. Or you might want to tell the query optimizer that you value the speed of returning the first row
more highly than total throughput. You can specify these and other behaviors by using query hints. The word hint is a
bit of a misnomer in this context because it is handled as a directive rather than as a suggestion. SQL Server provides
four general types of hints:
 Join hints, which specify the join technique that will be used.
 Index hints, which specify one or more specific indexes that should be used to drive a search or a sort.
 Lock hints, which specify a particular locking mode that should be used.
 Processing hints, which specify that a particular processing strategy should be used.

If you've made it this far in this book, you probably understand the various issues related to hints—locking, how
indexes are selected, overall throughput vs. the first row returned, and how join order is determined. Understanding
these issues is the key to effectively using hints and intelligently instructing SQL Server to deviate from "normal"
behavior when necessary. Hints are simply syntactic hooks that override default behavior; you should now have
insight into those behaviors so you can make good decisions about when such overrides are warranted.
This document is created with the unregistered version of CHM2PDF Pilot

Query hints should be used for special cases—not as standard operating procedure. When you specify a hint, you
constrain SQL Server. For example, if you indicate in a hint that a specific index should be used and later you add
another index that would be even more useful, the hint prevents the query optimizer from considering the new index.
In future versions of SQL Server, you can expect new query optimization strategies, more access methods, and new
locking behaviors. If you bind your queries to one specific behavior, you forgo your chances of benefiting from such
improvements. SQL Server offers the nonprocedural development approach—that is, you don't have to tell SQL
Server how to do something. Rather, you tell it what to do. Hints run contrary to this approach. Nonetheless, the
query optimizer isn't perfect and never can be. In some cases, hints can make the difference between a project's
success and failure, if they're used judiciously.

When you use a hint, you must have a clear vision of how it might help. It's a good idea to add a comment to the
query to justify the hint. Then you should test your hypothesis by watching the output of STATISTICS IO,
STATISTICS TIME, and one of the showplan options both with and without your hint.
TIP
Since your data is constantly changing, a hint that works well today might not indicate the appropriate processing
strategy next week. In addition, the query optimizer is constantly evolving, and the next upgrade or service pack you
apply might invalidate the need for that hint. You should periodically retest all queries that rely on hinting to verify that
the hint is still useful.

Join Hints
You can use join hints only when you use the ANSI-style syntax for joins—that is, when you actually use the word
JOIN in the query. In addition, the hint comes between the type of join and the word JOIN, so you can't leave off the
word INNER for an inner join. Here's an example of forcing SQL Server to use a HASH JOIN:
SELECT title_id, pub_name, title
FROM titles INNER HASH JOIN publishers
ON titles.pub_id = publishers.pub_id

Alternatively, you can specify a LOOP JOIN or a MERGE JOIN. You can use another join hint, REMOTE, when
you have defined a linked server and are doing a cross-server join. REMOTE specifies that the join operation is
performed on the site of the right table (that is, the table after the word JOIN). This is useful when the left table is a
local table and the right table is a remote table. You should use REMOTE only when the left table has fewer rows
than the right table.

Index Hints
You can specify that one or more specific indexes be used by naming them directly in the FROM clause. You can
also specify the indexes by their indid value, but this makes sense only when you specify not to use an index or to use
the clustered index, whatever it is. Otherwise, the indid value is prone to change as indexes are dropped and
re-created. You can use the value 0 to specify that no index should be used—that is, to force a table scan. And you
can use the value 1 to specify that the clustered index should be used regardless of the columns on which it exists. The
index hint syntax looks like this:

SELECT select_list
FROM table [(INDEX ({index_name | index_id}
[, index_name | index_id ...]))]

This example forces the query to do a table scan:

SELECT au_lname, au_fname
FROM authors (INDEX(0))
This document is created with the unregistered version of CHM2PDF Pilot

WHERE au_lname LIKE 'C%'

This example forces the query to use the index named aunmind:
SELECT au_lname, au_fname
FROM authors (INDEX(aunmind))
WHERE au_lname LIKE 'C%'

The following example forces the query to use both indexes 2 and 3. Note, however, that identifying an index by
indid is dangerous. If an index is dropped and re-created in a different place, it might take on a different indid. If you
don't have an index with a specified ID, you get an error message. The clustered index is the only index that will
always have the same indid value, which is 1. Also, you cannot specify index 0 (no index) along with any other
indexes.
SELECT au_lname, au_fname
FROM authors (INDEX (2,3))
WHERE au_lname LIKE 'C%'

We could use index hints in the example shown earlier, in which the query optimizer was not choosing to use an index
intersection. We could force the use of two indexes using this query:
SELECT fname, lname
FROM emps (INDEX (idx_nci_fname, idx_nci_lname))

A second kind of index hint is FASTFIRSTROW, which tells SQL Server to use a nonclustered index leaf level to
avoid sorting the data. This hint has been preserved only for backward compatibility; it has been superseded by the
processing hint FAST n, which is one of the query processing hints, as discussed in the next section.

Query Processing Hints

Query processing hints follow the word OPTION at the very end of your SQL statement and apply to the entire
query. If your query involves a UNION, only the last SELECT in the UNION can include the OPTION clause. You
can include more than one OPTION clause, but you can specify only one hint of each type. For example, you can
have one GROUP hint and also use the ROBUST PLAN hint. Here's an example of forcing SQL Server to process a
GROUP BY using hashing:
SELECT type, count(*)
FROM titles
GROUP BY type
OPTION (HASH GROUP)

This example uses multiple query processing hints:

SELECT pub_name, count(*)
FROM titles JOIN publishers
ON titles.pub_id = publishers.pub_id
GROUP BY pub_name
OPTION (ORDER GROUP, ROBUST PLAN, MERGE JOIN)

There are eight different types of processing hints, most of which are fully documented in SQL Server Books Online.
The key aspects of these hints are described here:

Grouping hints You can specify that SQL Server use a HASH GROUP or an ORDER GROUP to process
This document is created with the unregistered version of CHM2PDF Pilot

GROUP BY operations.

Union hints You can specify that SQL Server form the UNION of multiple result sets by using HASH UNION,
MERGE UNION, or CONCAT UNION. HASH UNION and MERGE UNION are ignored if the query specifies
UNION ALL because UNION ALL does not need to remove duplicates. UNION ALL is always carried out by
simple concatenation.

Join hints As discussed in the earlier section on join hints, you can specify join hints in the OPTION clause as well
as in the JOIN clause, and any join hint in the OPTION clause applies to all the joins in the query. OPTION join hints
override those in the JOIN clause. You can specify LOOP JOIN, HASH JOIN, or MERGE JOIN.

FAST number_rows This hint tells SQL Server to optimize so that the first rows come back as quickly as possible,
possibly reducing overall throughput. You can use this option to influence the query optimizer to drive a scan using a
nonclustered index that matches the ORDER BY clause of a query rather than using a different access method and
then doing a sort to match the ORDER BY clause. After the first number_rows are returned, the query continues
execution and produces its full result set.

FORCE ORDER This hint tells SQL Server to process the tables in exactly the order listed in your FROM clause.
However, if any of your joins is an OUTER JOIN, this hint might be ignored.

MAXDOP number This hint overrides the max degree of parallelism configuration option for only the query
specifying this option. I'll discuss parallelism when we look at server configuration in Chapter 17.

ROBUST PLAN This hint forces the query optimizer to attempt a plan that works for the maximum potential row
size, even if it means degrading performance. This is particularly applicable if you have wide varchar columns. When
the query is processed, some types of plans might create intermediate tables. Operators might need to store and
process rows that are wider than any of the input rows, which might exceed SQL Server's internal row size limit. If
this happens, SQL Server produces an error during query execution. If you use ROBUST PLAN, the query
optimizer will not consider any plans that might encounter this problem.

KEEP PLAN This hint ensures that a query is not recompiled as frequently when there are multiple updates to a
table. This can be particularly useful when a stored procedure does a lot of work with temporary tables, which might
cause frequent recompilations.

Stored Procedure Optimization

So far, I've been talking about tuning and optimizing individual queries. But what if you want to optimize an entire
stored procedure? In one sense, there's nothing special about optimizing a stored procedure because each statement
inside the procedure is optimized individually. If every statement has good indexes to use, and each one performs
optimally, the procedure should also perform optimally. However, since the execution plan for a stored procedure is a
single unit, there are some additional issues to be aware of. The main issue is how often the entire procedure is
recompiled. In Chapter 11, we looked at how and when you might want to force a procedure to be recompiled,
either on an occasional basis or every time it is executed. I also mentioned some situations in which SQL Server will
automatically recompile a stored procedure.

For a complex procedure or function, recompilation can take a substantial amount of time. As I've mentioned, in
some cases you must recompile the entire procedure to get the optimum plan, but you don't want the procedure to be
recompiled more often than necessary. Tuning a stored procedure, as opposed to tuning individual queries, involves
making sure that the procedure is not recompiled more often than it needs to be. In fact, a badly tuned stored
procedure can end up being recompiled numerous times for every single execution.

In this section, I'll mention three specific situations that can cause a run-time recompilation of a stored procedure or
function:
 Data in a table referenced by the routine has changed.
This document is created with the unregistered version of CHM2PDF Pilot

 The procedure contains a mixture of DDL and DML operations.

 Certain operations on temporary tables are performed.

These situations are actually discussed in detail in Microsoft Knowledge Base article Q243586, which is included on
the companion CD. In this section, I'll just mention the most important issues.

Data Changing in a Referenced Table

You saw in Chapter 15 that SQL Server 2000 will automatically update index statistics if more than a certain number
of row modification operations have been performed on a table. And, if any table referenced by a procedure has had
its statistics updated, the procedure will automatically be updated the next time it is executed. This is particularly
common when your procedure creates a temporary table. When the table is first created, it has no rows in it, so any
added rows can invoke an update of statistics and thus a recompile. In fact, query plans involving temporary tables
are more likely to be recompiled than query plans involving permanent tables because only six modification operations
have to occur in order to force an update of statistics.

If you find that your stored procedures are being recompiled more often than you think is necessary, you can use the
KEEP PLAN query hint. Note that the KEEP PLAN hint applies only to recompiling a procedure due to row
modifications for a temporary table.

You can use SQL Profiler to determine if and when your procedures are being recompiled. (Knowledge Base article
Q243586 supplies a lot of detail on using SQL Profiler.)

Mixing DDL and DML

If any DDL operations are performed, either on a temporary table or a permanent table, the procedure is always
recompiled as soon as another DML operation is encountered. This situation is called interleaving of DDL and
DML. DDL operations can include creating a table, modifying a table, adding or changing a table constraint, or
adding an index to a table. If you have a procedure that creates a table, selects from it, creates an index, and then
selects from the table again, the procedure will be recompiled twice while it is running. And this doesn't take into
account the fact that it might have been compiled before it ever started execution the first time, perhaps due to not
having a plan available in cache.

To avoid this situation, you should have all the DDL right at the beginning of the procedure so that there is no
interleaving. The procedure will then have to be recompiled only once, while it is executing.

Other Operations on Temporary Tables

Certain other operations on temporary tables will cause recompiles even though these same operations on a
permanent table might not cause a procedure to be recompiled. You can minimize these recompiles by making sure
that your procedures meet the following conditions:
 All references to a temporary table refer to a table created locally, not to temporary tables created in the
calling procedure or batch.
 The procedure does not declare any cursors that reference a temporary table.
 The procedure does not create temporary tables inside a conditionally executed statement such as IF/ELSE
or WHILE.
This document is created with the unregistered version of CHM2PDF Pilot

34

Concurrency and Consistency

Tradeoffs
If you're designing a multiuser application that will both query and modify the same data, you need a good
understanding of concurrency and consistency. Concurrency is the ability to have many simultaneous users operating
at once. The more users who can work well simultaneously, the higher your concurrency. Consistency is the level at
which the data in multiuser scenarios exhibits the same behavior that it would if only one user were operating at a time.
Consistency is expressed in terms of isolation levels. At the highest isolation level, Serializable, the multiuser system
behaves just as it would if users submitted their requests serially. At the Serializable level you have a greater need to
protect resources, which you do by locking, which in turn reduces concurrency. (Locking is discussed in Chapter 14.
You should also understand transactional concepts, which are covered in Chapter 12.)

In many typical environments, an ongoing struggle occurs between the OLTP demands on the database and the DSS
demands. OLTP is characterized by relatively high volumes of transactions that modify data. The transactions tend to
be relatively short and usually don't query large amounts of data. DSS is read intensive, often with complex queries
that can take a long time to complete and thus hold locks for a long time. The exclusive locks needed for data
modification with OLTP applications block the shared locks used for DSS. And DSS tends to use many shared locks
and often holds them for long periods and thus stalls the OLTP applications, which must wait to acquire the exclusive
locks required for updating. DSS also tends to benefit from many indexes and often from a denormalized database
design that reduces the number of tables to be joined. Large numbers of indexes are a drag on OLTP because of the
additional work to keep them updated. And denormalization means redundancy—a tax on the update procedures.

You need to understand the isolation level required by your application. Certainly, you do not want to request
Serializable (or, equivalently, use the HOLDLOCK or SERIALIZABLE hint) if you need only Read Committed. And
it might become clear that some queries in your application require only Read Uncommitted (dirty read) because they
look for trends and don't need guaranteed precision. If that's the case, you might have some flexibility in terms of
queries not requiring shared locks, which keeps them from being blocked by processes modifying the database and
from blocking those modifying processes. But even if you can live with a dirty-read level, you should use it only if
necessary. This isolation level means that you might read data that logically never existed; this can create some weird
conditions in the application. For example, a row that's just been read will seem to vanish because it gets rolled back.
If your application can live with a dirty-read isolation level, it might be comforting to have that as a fallback. However,
you should probably first try to make your application work with the standard isolation level of Read Committed and
fall back to dirty read only if necessary.
This document is created with the unregistered version of CHM2PDF Pilot

34

Resolving Blocking Problems

Many applications suffer from poor performance because processes are backed up waiting to acquire locks or
because of deadlocks. A smooth, fast application should minimize the time spent waiting for locks, and it should avoid
deadlocks. The most important step you can take is to first understand how locking works.

A process blocks when it stalls while waiting to acquire a lock that is incompatible with a lock held by some other
process. This condition is often, but erroneously, referred to as a "deadlock." As long as the process being stalled is
not, in turn, stalling the offending process—which results in a circular chain that will never work itself out without
intervention—you have a blocking problem, not a deadlock. If the blocking process is simply holding on to locks or is
itself blocked by some other process, this is not a deadlock either. The process requesting the lock must wait for the
other process to release the incompatible lock; when it does, all will be fine. Of course, if the process holds that lock
excessively, performance still grinds to a halt and you must deal with the blocking problem. Your process will suffer
from bad performance and might also hold locks that stall other processes; every system on the network will appear
to hang.

The following guidelines will help you avoid or resolve blocking problems:
 Keep transactions as short as possible. Ideally, a BEGIN TRAN?COMMIT TRAN block should include
only the actual DML statements that must be included in the explicit transaction. As much as possible, do
conditional logic, variable assignment, and other "setup" work before the BEGIN TRAN. The shorter the
duration of the transaction, the shorter the time that locks will be held. Keep the entire transaction within one
batch if possible.
 Never add a pause within a transaction for user input. This rule is basically a part of the previous one, but it is
especially important. Humans are slow and unreliable compared to computers. Do not add a pause in the
middle of the transaction to ask a user for input or to confirm some action. The person might decide to get up
to take a coffee break, stalling the transaction and making it hold locks that cause blocking problems for other
processes. If some locks must be held until the user provides more information, you should set timers in your
application so that even if the user decides to go to lunch, the transaction will be aborted and the locks will be
released. Similarly, give your applications a way to cancel out of a query if such an action is necessary.
Alternatively, you can use the LOCK_TIMEOUT session option to control when SQL Server automatically
cancels out of a locking situation. We'll look at this option later in this chapter.
 When you process a result set, process all rows as quickly as possible. Recall from Chapter 3 that an
application that stops processing results can prevent the server from sending more results and stall the
scanning process, which requires locks to be held much longer.
 For browsing applications, consider using cursors with optimistic concurrency control. An address book
application is a good example of a browsing application: users scroll around to look at data and occasionally
update it. But the update activity is relatively infrequent compared to the time spent perusing the data. Using
scrollable cursors with the OPTIMISTIC locking mode is a good solution for such applications. Instead of
locking, the cursor's optimistic concurrency logic determines whether the row has changed from the copy that
your cursor read. If the row has not changed during the lengthy period in which the user is perusing the data,
the update is made without holding locks. If the row has changed, the UPDATE statement produces an error
and the application can decide how to respond. Although I strenuously cautioned you against the misuse of
cursors in Chapter 13, they are ideally suited to browsing applications.

You can also easily implement your own optimistic locking mechanism without using cursors. Save the values
of the data you selected and add a WHERE clause to your update that checks whether the values in the
current data are the same as those you retrieved. Or, rather than use the values of the data, use a SQL
Server rowversion column—an ever-increasing number that's updated whenever the row is touched,
unrelated to the system time. If the values or rowversion are not identical, your update will not find any
qualifying row and will not affect anything. You can also detect changes with @@ROWCOUNT and decide
This document is created with the unregistered version of CHM2PDF Pilot

to simply abort, or more typically, you can indicate to the user that the values have changed and then ask
whether the update should still be performed. But between the time the data was initially retrieved and the
time the update request was issued, shared locks are not held, so the likelihood of blocking and deadlocks is
significantly reduced.

Indexes and Blocking

I've already recommended that you choose your indexes wisely, strictly for performance reasons. However,
concurrency concerns are also a reason to make sure you have good indexes on your tables. (Of course, better
concurrency can also lead to better performance.) You saw in Chapter 14 that SQL Server acquires row (or key)
locks whenever possible. However, this does not always mean that no other rows are affected if you're updating only
one row in a table. Remember that to find the row to update, SQL Server must first do a search and acquire
UPDATE locks on the resources it inspects. If SQL Server does not have a useful index to help find the desired row,
it uses a table scan. This means every row in the table acquires an update lock, and the row actually being updated
acquires an exclusive lock, which is not released until the end of the transaction. Here's a small example that illustrates
the crucial relationship between indexes and concurrency:
USE pubs
GO
DROP TABLE t1
GO
/* First create and populate a small table. */
CREATE TABLE t1 (a int)
GO
INSERT INTO t1 VALUES (1)
INSERT INTO t1 VALUES (3)
INSERT INTO t1 VALUES (5)
GO

BEGIN tran
UPDATE t1
SET a = 7 WHERE a = 1

EXEC sp_lock @@spid

/* The output here should show you one X lock, on a RID; that is the
row that has been updated. */

/* In another query window, run this batch before rollback is

issued: */

USE pubs
UPDATE t1
SET a = 10
WHERE a = 3

/* Execute the rollback in the first window. */

ROLLBACK TRAN

You should have noticed that the second connection was unable to proceed. To find the row where a = 3, it tries to
scan the table, first acquiring update locks. However, it cannot obtain an update lock on the first row, which now has
a value of 7 for a, because that row is exclusively locked. Since SQL Server has no way to know whether that is a
row it is looking for without being able to look at it, this second connection blocks. When the rollback occurs in the
first connection, the locks are released and the second connection can finish.

Let's see what happens if we put an index on the table. We'll run the same script again, except this time we'll build a
nonclustered index on column a:
USE pubs
This document is created with the unregistered version of CHM2PDF Pilot

GO
DROP TABLE t1
GO
/* First create and populate a small table. */
CREATE TABLE t1 ( a int)
GO
CREATE INDEX idx1 ON t1(a)
GO
INSERT INTO t1 VALUES (1)
INSERT INTO t1 VALUES (3)
INSERT INTO t1 VALUES (5)
GO

BEGIN tran
UPDATE t1
SET a = 7 WHERE a = 1

EXEC sp_lock @@spid

/* In this case, the output should show you three X locks (one again on
a RID) and two KEY locks. When the key column a is changed, the leaf
level of the nonclustered index is adjusted. Since the leaf level of
the index keeps all the keys in sorted order, the old key with the
value 1 is moved from the beginning of the leaf level to the end,
because now its value is 7. However, until the transaction is over, a
ghost entry is left in the original position and the key lock is
maintained. So there are two key locks: one for the old value and one
for the new. */

/* In another query window, run this batch before rollback is issued: */

USE pubs
UPDATE t1
SET a = 10
WHERE a = 3

/* Execute the rollback in the first window. */

ROLLBACK TRAN

This time, the second query succeeded, even though the first query held X locks on the keys in the leaf level of the
index. The second connection was able to generate the lock resource string for the keys it needed to lock and then
request locks only on those particular keys. Since the keys that the second connection requested were not the same
as the keys that the first connection locked, there was no conflict over locking resources and the second connection
could proceed.
This document is created with the unregistered version of CHM2PDF Pilot

34

Resolving Deadlock Problems

Deadlocks befuddle many programmers and are the bane of many applications. A deadlock occurs when, without
some intervening action, processes cannot get the locks they need no matter how long they wait. Simply waiting for
locks is not a deadlock condition. SQL Server automatically detects the deadlock condition and terminates one of the
processes to resolve the situation. The process gets the infamous error message 1205 indicating that it was selected as
the "victim" and that its batch and the current transaction have been canceled. The other process can then get the
locks it needs and proceed. The two general forms of deadlocks are cycle deadlocks and conversion deadlocks.
(See Chapter 14 for more on these two kinds of deadlocks.)

Cycle Deadlock Example

If you repeatedly run the following two transactions simultaneously from different Osql.EXE sessions, you're nearly
assured of encountering a "deadly embrace" (cycle deadlock) almost immediately from one of the two processes. It
should be clear why: one of the processes gets an exclusive lock on a row in the authors table and needs an exclusive
lock on a row in the employee table. The other process gets an exclusive lock on the row in employee, but it needs
an exclusive lock for the same row in authors that the first process has locked.
-- Connection 1
USE pubs
WHILE (1=1)
BEGIN
BEGIN TRAN
UPDATE employee SET lname='Smith' WHERE emp_id='PMA42628M'
UPDATE authors SET au_lname='Jones' WHERE au_id='172-32-1176'
COMMIT TRAN
END

-- Connection 2
USE pubs
WHILE (1=1)
BEGIN
BEGIN TRAN
UPDATE authors SET au_lname='Jones' WHERE au_id='172-32-1176'
UPDATE employee SET lname='Smith' WHERE emp_id='PMA42628M'
COMMIT TRAN
END

The result is the dreaded error 1205 (shown here) from one of the connections. The other connection continues along
as if no problem has occurred.
Server: Msg 1205, Level 13, State 50, Line 1
Transaction (Process ID 52) was deadlocked on {lock} resources with
another process and has been chosen as the deadlock victim. Rerun the
transaction.

If you simply rewrite one of the batches so that both batches first update authors and then update employee, the two
connections will run forever without falling into a cycle deadlock. Or you can first update employee and then update
authors from both connections. Which table you update first doesn't matter, but the updates must be consistent and
you must follow a known protocol. If your application design specifies a standard protocol for accessing the tables
consistently, one of the connections gets the exclusive page lock on the first table and the other process must wait for
This document is created with the unregistered version of CHM2PDF Pilot

the lock to be released. Simply waiting momentarily for a lock is normal and usually fast, and it happens frequently
without your realizing it. It is not a deadlock.

Conversion Deadlock Example

Now run the following transaction simultaneously from two different OSQL.EXE or SQL Query Analyzer sessions.
You're running the same script, so it's obvious that the two processes follow a consistent order for accessing tables.
But this example quickly produces a deadlock. A delay has been added so that you encounter the race condition
more quickly, but the condition is lurking there even without the WAITFOR DELAY—the delay just widens the
window.
USE pubs
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRAN
SELECT * FROM authors WHERE au_id='172-32-1176'
-- Add 5 sec sleep to widen the window for the deadlock
WAITFOR DELAY '00:00:05'
UPDATE authors SET au_lname='Updated by '
+ CONVERT(varchar, @@spid) WHERE au_id='172-32-1176'
COMMIT TRAN

You can correct this example in a couple of ways. Does the isolation level need to be Repeatable Read? If Read
Committed is sufficient, you can simply change the isolation level to get rid of the deadlock and to provide better
concurrency. If you use Read Committed isolation, the shared locks can be released after the SELECT, and then one
of the two processes can acquire the exclusive lock it needs. The other process waits for the exclusive lock and
acquires it as soon as the first process finishes its update. All operations progress smoothly, and the queuing and
waiting happen invisibly and so quickly that it is not noticeable to the processes.

But suppose that you need Repeatable Read (or Serializable) isolation. In this case, you should serialize access by
using an update lock, which you request using the UPDLOCK hint. Recall from Chapter 14 that an update lock is
compatible with a shared lock for the same resource. But two update locks for the same resource are not compatible,
and update and exclusive locks are also not compatible. Acquiring an update lock does not prevent others from
reading the same data, but it does ensure that you're first in line to upgrade your lock to an exclusive lock if you
subsequently decide to modify the locked data. (The important issue here is that the exclusive lock can be acquired,
so this technique works equally well even if that second statement is a DELETE and not an UPDATE, as in this
example.)

By serializing access to the exclusive lock, you prevent the deadlock situation. The serialization also reduces
concurrency, but that's the price you must pay to achieve the high level of transaction isolation. A modified example
follows; multiple simultaneous instances of this example will not deadlock. Try running about 20 simultaneous
instances and see for yourself. None of the instances deadlocks and all complete, but they run serially. With the
built-in 5-second sleep, it takes about 100 seconds for all 20 connections to complete because one connection
completes about every 5 seconds. This illustrates the lower concurrency that results from the need for higher levels of
consistency (in this case, Repeatable Read).
USE pubs
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRAN
SELECT * FROM authors (UPDLOCK) WHERE au_id='172-32-1176'
-- Add 5 sec sleep to widen the window for the deadlock
WAITFOR DELAY '00:00:05'
UPDATE authors SET au_lname='Updated by '
+ CONVERT(varchar, @@spid) WHERE au_id='172-32-1176'
COMMIT TRAN
This document is created with the unregistered version of CHM2PDF Pilot

As a general strategy, you should add the UPDLOCK hint (or some other serialization) if you discover during testing
that conversion deadlocks are occurring. Or add UPDLOCK from the outset if you detect from your CRUD analysis
that deadlocks are likely, and then try backing it off during multiuser testing to see if it is absolutely necessary. Either
approach is viable. If you know that in most cases the read-for-update (discussed later) will follow with an actual
update, you might opt for the UPDLOCK hint from the outset and see if you can back it off later. But if the
transaction is short and you will later update the data in most cases, there isn't much point in backing off the update
lock. The shared lock will need to upgrade to an exclusive lock, so getting the update lock in the first place makes
good sense.

Preventing Deadlocks
Deadlocks can usually be avoided, although you might have to do some detailed analysis to solve the problems that
cause them. Sometimes the cure is worse than the ailment, and you're better off handling deadlocks rather than totally
preventing them, as we'll discuss in the next section. Preventing deadlocks (especially conversion deadlocks) requires
a thorough understanding of lock compatibility. These are the main techniques you can use to prevent deadlocks:
 To prevent cycle deadlocks, make all processes access resources in a consistent order.
 Reduce the transaction isolation level if it's suitable for the application.
 To prevent conversion deadlocks, explicitly serialize access to a resource.

Deadlock prevention is a good reason to use stored procedures. By encapsulating the data access logic in stored
procedures, it's easier to impose consistent protocols for the order in which resources (for example, tables) are
accessed, which can help you avoid cycle deadlocks. But in so doing, you do not reduce the likelihood of conversion
deadlocks. As noted above, conversion deadlocks are best dealt with by serializing access to a resource or by
lowering the transaction isolation level if appropriate. The most common scenario for conversion deadlocks is the
read-for-update situation. If a resource will be read within a transaction requiring Repeatable Read or Serializable
isolation and will be updated later, you should request an update lock on the read using the UPDLOCK hint. The
update lock will let other users read the data but will prevent them from updating the data or doing a read-for-update.
The net effect is that once the update lock is acquired, the process will be next in line for the exclusive lock needed to
actually modify it.

Handling Deadlocks
The cost of serializing access is that other users wanting to read-for-update (or actually update or delete) must wait.
If you are not experiencing deadlocks, serializing access might needlessly reduce concurrency. You might have a case
in which a transaction often does a read-for-update but only infrequently does the update. In this case, deadlocks
might be infrequent. The best course of action might be to not prevent deadlocks and simply deal with them when
they occur.

Preventing deadlocks can significantly reduce concurrency because the read-for-update would be blocked. Instead,
you can simply write your applications to handle deadlocking. Check for deadlock message 1205, and retry the
transaction. With retry logic, you can live with moderate deadlock activity without adding a lot of serialization to the
application. It's still a good idea to keep count of how often you experience deadlocks; if the incidence is high, the
wasted effort and constant retrying are likely to be worse than the cost of preventing the deadlock in the first place.
How you write the deadlock handler will depend on the language or tool you use to build your application. But an
application that is prone to deadlocks should have a deadlock handler that retries. Such a handler must be written in
the host language. There is no way to do a retry directly within a stored procedure or a batch because deadlock error
1205 terminates the batch.

Volunteering to Be the Deadlock Victim

Recall from Chapter 14 that the LOCK_MONITOR process in SQL Server attempts to choose as the deadlock
victim the process that would be the least expensive to roll back, considering the amount of work the process has
This document is created with the unregistered version of CHM2PDF Pilot

already done. But a process can also offer to "sacrifice itself" as the victim for deadlock resolution. You can make this
happen by using the SET DEADLOCK_PRIORITY LOW | NORMAL statement. If a process has a deadlock priority
of LOW and the other participating process is NORMAL (the default), the LOW process is chosen as the victim
even if it was not the process that closed the loop.

Adding SET DEADLOCK_PRIORITY LOW to one of the connections (and not the other) indicates that it will be
selected as the victim, even if it will be more expensive to roll back. You might find this useful if, for example, you are
building summary tables for reporting purposes, and also performing OLTP activities on the same database and you
occasionally have deadlocks and you know that one process is more important than the other. You set the less
important process to LOW. It might also be useful if one application was written with a good deadlock handler and
the other was not. Until the application without the handler can be fixed, the "good" application can make the simple
change of volunteering itself and then handle a deadlock with retry logic when the deadlock occurs later.

Watching Locking Activity

Locking problems often result from locks that you don't even realize are being taken. You might be updating only
table A, but blocking issues arise on table B because of relationships that you don't realize exist. If, for example, a
foreign key relationship exists between table A and table B and the update on A is causing some query activity to B,
some shared locks must exist. Or a trigger or a nested call to another procedure might cause some locking that isn't
obvious to you.

In cases like these, you must be able to watch locks to look for locking operations that you weren't aware of. The
graphical lock display of SQL Server Enterprise Manager is the most convenient way to watch locking in many cases.
However, SQL Server Enterprise Manager provides only a snapshot of the current state of locking; you can miss a
lot of locking activity that occurs in the time it takes you to refresh the display.

Identifying the Culprit

As with most debugging situations, the hardest part of solving a problem with locking is understanding the problem. If
you get complaints that "the system is hung," it's a good bet that you have a blocking problem. Most blocking
problems happen because a single process holds locks for an extended period of time. A classic case (as I discussed
earlier) is an interactive application that holds locks until the user at the other end takes an action, such as clicking a
button to commit the transaction or scrolling to the end of the output, which causes all the results to be processed. If
the user goes to lunch and doesn't take that action, everyone else might as well grab lunch too, because they're not
going to get much work done. Locks pile up, and the system seems to hang. The locks in the system are reported in
the pseudo-system table, syslockinfo in the master database. I'll talk about a related pseudo-system table,
sysprocesses, later in the chapter.
NOTE
The pseudo-system tables are not maintained as on-disk structures. Locking and process data are by definition
relevant only at runtime. So syslockinfo and sysprocesses are presented as system tables, and although they can be
queried just like other system tables, they do not have on-disk storage as normal tables do.

A big challenge in trying to identify the cause of blocking is that when you experience such problems, you probably
have hundreds or thousands of locks piled up. At these times, it's hard to see the trees through the forest. Usually, you
just want to see which process is holding up everyone else. Once you identify it, of course, you need to immediately
get it out of the way, either by forcing it to complete or, if necessary, by issuing the KILL command on the
connection. The longer-term solution is to rework the application so that it will not hold locks indefinitely.

SQL Server Enterprise Manager provides a graphical way to watch locking activity, as shown in Figure 16-6; this is
the best method most of the time. From the Management folder, choose Current Activity. You can look at locks
either by process or by object. If you select the Locks / Process ID option, the graphic shows which processes are
blocked and which processes are blocking. When you double-click on the process in the right pane, you see the last
This document is created with the unregistered version of CHM2PDF Pilot

command the process issued. This is what Figure 16-6 shows. You can then send a message to the offending user to
complete the transaction (although this is a short-term solution at best). Or you can kill the process from SQL Server
Enterprise Manager, which isn't ideal but is sometimes the best short-term course of action. You can select a
particular process from the left pane, under the Locks / Process ID option, and the right pane will show all the locks
held by that process.

Figure 16-6. Viewing the last batch issued by a blocking process in SQL Server Enterprise Manager.

Alternatively, you can select the Locks / Object option under Current Activity. The left pane displays a graphic of
each locked object. If you then select one of these objects, the right pane shows which processes have that object
locked. Again, when you double-click on the process in the right pane, you see the last command the process issued.

But sometimes even SQL Server Enterprise Manager can get blocked by locking activity in tempdb, so it's useful to
know how to monitor-lock the old-fashioned way, by using system stored procedures or by querying directly from
the syslockinfo table. You start by running sp_who2 and sp_lock. Most people are familiar with sp_who but not
sp_who2, which works in almost the same way but formats the output in a more readable way and contains more of
the columns from the sysprocesses table. However, sysprocesses contains additional information that neither sp_who
nor sp_who2 reports, so you might even want to write your own sp_who3! We'll look at the sysprocesses table in
more detail later in this chapter.

The BlkBy column of the sp_who2 output shows the ID (spid) of a blocking process. The procedure sp_lock
provides a formatted and sorted listing of syslockinfo that decodes the lock types into mnemonic forms ,such as IS
(Intent-Shared) instead of type 7. In Chapter 14, we looked at output from sp_lock to watch the various types and
modes of locking. If your users say that the system is hung, try to log on and execute sp_who2 and sp_lock. If you
can log on and execute these procedures, you immediately know that the system is not hung. If you see a nonzero
spid in the BlkBy column of the sp_who2 output or if the Status value is WAIT in the sp_lock output, blocking is
occurring. It's normal for some blocking to occur—it simply indicates that one process is waiting for a resource held
by another. If such contention didn't exist, you wouldn't even need to lock. But if you reissue the query for lock
activity a moment later, you expect to find that same blockage cleared up. In a smoothly running system, the duration
that locks are held is short, so long pileups of locks don't occur.

When a major lock pileup occurs, you can get a lengthy chain of processes blocking other processes. It can get
pretty cumbersome to try to track this manually. You can create a procedure like the following to look for the process
at the head of a blocking chain:
USE master
GO
CREATE PROCEDURE sp_leadblocker
This document is created with the unregistered version of CHM2PDF Pilot

AS
IF EXISTS
(SELECT * FROM master.dbo.sysprocesses
WHERE spid IN (SELECT blocked FROM master.dbo.sysprocesses))
SELECT spid, status, loginame=SUBSTRING(loginame,1,12),
hostname=substring(hostname, 1, 12),
blk=CONVERT(char(3), blocked),
dbname=SUBSTRING(DB_NAME(dbid), 1, 10), cmd, waittype
FROM master.dbo.sysprocesses
WHERE spid IN (SELECT blocked FROM master.dbo.sysprocesses)
AND blocked=0
ELSE
SELECT 'No blocking processes found!'

Once you identify the connection (spid) causing the problem, check to see the specific locks that it is holding. You
can query from syslockinfo for this, but simply running sp_lock spid will probably give you exactly what you need.
There might be multiple separate lock chains, in which case the batch will return more than one spid. You can follow
the same procedure for each one.

The syslockinfo Table

If a lot of blocking is going on, you might want to take a look at all locks held that are blocking other users. You can
use the procedure sp_blockinglocks (which is not a part of the installed SQL Server product but is included on the
companion CD) to print out a list of all locks held that are blocking other processes. The procedure examines the
syslockinfo table for resources that have more than one lock listed and are held by different processes with different
status values. For example, if process 52 has an X lock on the sales table with a status of GRANT and process 53 is
trying to read from that table, the output of sp_blockinglocks looks like this:
spid db ObjId IndId Type Resource Mode Status
------ ----- ----------- ------ ---- ---------------- -------- ------
52 pubs 149575571 0 TAB X GRANT
53 pubs 149575571 0 TAB IS WAIT

It can also be useful to see the last command issued by a blocking process; you can do this by double-clicking on a
process ID in the Current Activity panes of SQL Server Enterprise Manager. You can also use DBCC
INPUTBUFFER (spid) for this—which is the same thing SQL Server Enterprise Manager does when you
double-click on a process. DBCC INPUTBUFFER reads the memory from within the server that was used for the
last command. So DBCC INPUTBUFFER can access whatever command is still in the buffer for a given connection,
even if it has already been executed.

Usually, by this point you have enough of a handle on the problem to turn your attention to the application, which is
where the resolution will ultimately lie. But if you need or want to go a bit further, you can find out the depth at which
the connection is nested within a transaction. Blocking problems often happen because the connection doesn't realize
the nested depth and hasn't applied the correct pairing of COMMIT commands. (Chapter 12 discusses the scoping
and nesting levels of transactions.) A connection can determine its own nesting depth by querying
@@TRANCOUNT. Starting with SQL Server 7, you can also determine the nesting level of any current connection,
not just your own. This involves inspecting the sysprocesses table directly because the information is not included in
either sp_who or sp_who2.

The sysprocesses Table

As I've mentioned, some columns in the sysprocesses table don't show up in the output of either sp_who or
sp_who2. Table 16-2 describes these columns.

Table 16-2. Columns in the sysprocesses table that aren't available via sp_who or sp_who2.
This document is created with the unregistered version of CHM2PDF Pilot

Column Name Data Type Description

The waittypes of 1 through 15
correspond to the mode of lock being
waited on. Hex values 0x40 through
0x4f correspond to miscellaneous
waittype binary(2)
waits. 0x81 is a wait to write to the
log. 0x400 through 0x40f are latch
waits. 0x800 is a wait on a network
write.
Indicates the current wait time in
waittime int milliseconds. The value is 0 when the
process is not waiting.
A string indicating the name of the last
lastwaittype nchar(32)
or current wait type.
A textual representation of a lock
waitresource nchar(32)
resource.
Indicates the number of pages in the
memory cache that are currently
allocated to this process. A negative
memusage int
number means that the process is
freeing memory allocated by another
process.
Indicates the time when a client
process logged on to the server. For
login_time datetime
system processes, it indicates the time
when SQL Server startup occurred.
An execution context ID that uniquely
ecid smallint identifies the subthreads operating on
behalf of a single process.
kpid smallint The operating system thread ID.
Indicates the current transaction
open_tran smallint
nesting depth for the process.
The user ID for the user who
uid smallint
executed the command.
A globally unique identifier (GUID)
sid binary(85)
for the login.
The operating system domain for the
nt_domain nchar(128)
client.
The operating system username for
nt_username nchar(128)
the process.
An assigned unique identifier for the
net_address nchar(12) network interface card on each user's
workstation.
The name of the client's network
net_library nchar(12)
library DLL.

The procedure sp_who2 translates the dbid into a database name and translates the numeric value for its status into a
string. The most common status values you'll see are BACKGROUND, ROLLBACK, DORMANT (used when a
connection is being retained while waiting for Remote Procedure Call [RPC] processing), SLEEPING, and
RUNNABLE.
This document is created with the unregistered version of CHM2PDF Pilot

The values that the sysprocesses table holds in the waittype column are the same ones you'll see in the req_mode
column of syslockinfo. They can be translated as follows:
Lock Mode value
----------- -------
NULL 1
Sch-S 2
Sch-M 3
S 4
U 5
X 6
IS 7
IU 8
IX 9
SIU 10
SIX 11
UIX 12
BU 13
RangeS-S 14
RangeS-U 15
RangeIn-Nul 16
RangeIn-S 17
RangeIn-U 18
RangeIn-X 19
RangeX-S 20
RangeX-U 21
RangeX-X 22

NOTE
These values are not the same as the values shown in SQL Server Books Online under the description for the
syslockinfo table. The values in SQL Server Books Online are incorrect.

The Trace Flag for Analyzing Deadlock Activity

Trace flags are useful for analyzing deadlock situations. When a process is part of a deadlock, the victim process
realizes that the deadlock occurred when it gets error message 1205. Any other process participating in the deadlock
is unaware of the situation. To resolve the deadlock, you probably want to see both processes; trace flag 1204
provides this information.

The following is a fragment of the output from sqlservr.exe, which was started from the command line using –T1204.
This example uses the conversion deadlock script that we used previously—issuing the same batch from two
connections—to illustrate the output of trace flag 1204. To capture deadlock information, you must start SQL Server
from a command prompt; the output of the trace flag will be displayed in the command window. If you're running a
named instance of SQL Server 2000, you must be in the binn directory for that instance (such as
\mssql$MyInstanceName\binn), and you must also supply the instance name with the -s parameter.
sqlservr _c _T1204 -s MyInstance
Deadlock encountered .... Printing deadlock information
Wait-for graph
Node:1
KEY: 5:1977058079:1 (02014f0bec4e) CleanCnt:2 Mode: S Flags: 0x0
Grant List::
Owner:0x192713e0 Mode: S Flg:0x0 Ref:1 Life:02000000 SPID:52 ECID:0
SPID: 52 ECID: 0 Statement Type: UPDATE Line #: 7
Input Buf: Language Event: USE pubs
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRAN
SELECT * FROM authors WHERE au_id='172-32-1176'
-- Add 5 sec sleep to widen the window for the deadlock
WAITFOR DELAY '00:00:05'
This document is created with the unregistered version of CHM2PDF Pilot

UPDATE authors SET au_lname='Updated by '

+ CO
Requested By:
ResType:LockOwner Stype:'OR' Mode: X SPID:51 ECID:0 Ec:(0x192794e8)
Value:0x19271400 Cost:(0/0)

Node:2
KEY: 5:1977058079:1 (02014f0bec4e) CleanCnt:2 Mode: S Flags: 0x0
Grant List::
Owner:0x192712e0 Mode: S Flg:0x0 Ref:1 Life:02000000 SPID:51 ECID:0
SPID: 51 ECID: 0 Statement Type: UPDATE Line #: 7
Input Buf: Language Event: USE pubs
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
BEGIN TRAN
SELECT * FROM authors WHERE au_id='172-32-1176'
-- Add 5 sec sleep to widen the window for the deadlock
WAITFOR DELAY '00:00:05'
UPDATE authors SET au_lname='Updated by '
+ CO
Requested By:
ResType:LockOwner Stype:'OR' Mode: X SPID:52 ECID:0 Ec:(0x192c34e8)
Value:0x1926d400 Cost:(0/0)
Victim Resource Owner:
ResType:LockOwner Stype:'OR' Mode: X SPID:52 ECID:0 Ec:(0x192c34e8)
Value:0x1926d400 Cost:(0/0)

This output shows the spid for both processes affected, shows a fragment of their input buffer (but not the entire
command), and notes that neither process can upgrade its locks because of the circular relationship. Process 52
requests an update lock on a key that process 51 already has an update lock on. At the same time, process 51
requests an exclusive lock on a key that process 52 already has a shared lock on. Thus, the two processes are in a
circular relationship. The output can help you solve the problem—you know the processes involved and the locks that
could not be acquired, and you have an idea of the commands being executed.
NOTE
The number assigned to trace flag 1204 is intended to be easy to remember. Recall that error 1205 is the
well-known error message that an application receives when it is chosen as the deadlock victim.

Lock Hints
You can specify a lock type or duration with syntax similar to that for specifying index hints. Chapter 14 explains lock
compatibility and other issues that are essential to using lock hints properly. Lock hints work only in the context of a
transaction, so if you use these hints, you must use BEGIN TRAN/COMMIT (or ROLLBACK) TRAN blocks (or
you must run with implicit transactions set to ON). The lock hint syntax is as follows:

SELECT select_list
FROM table_name [(lock_type)]

TIP
Remember to put the lock hint in parentheses; if you don't, it will be treated as an alias name for the table instead of
as a lock hint.

You can specify one of the following keywords for lock_type:

HOLDLOCK
This is equivalent to the SERIALIZABLE hint described below. This option is similar to specifying SET
TRANSACTION ISOLATION LEVEL SERIALIZABLE , except that the SET option affects all tables, not only
This document is created with the unregistered version of CHM2PDF Pilot

the one specified in this hint.

UPDLOCK
This takes update page locks instead of shared page locks while reading the table and holds them until the end of the
transaction. Taking update locks can be an important technique for eliminating conversion deadlocks.

TABLOCK
This takes a shared lock on the table even if page locks would be taken otherwise. This option is useful when you
know you'll escalate to a table lock or if you need to get a complete snapshot of a table. You can use this option with
holdlock if you want the table lock held until the end of the transaction block (repeatable read).

PAGLOCK
This keyword takes shared page locks when a single shared table lock might otherwise be taken. (To request an
exclusive page lock you must use the XLOCK hint along with the PAGLOCK hint.)

TABLOCKX
This takes an exclusive lock on the table that is held until the end of the transaction block. (All exclusive locks are
held until the end of a transaction, regardless of the isolation level in effect. This hint has the same effect as specifying
both the TABLOCK and the XLOCK hints together. )

ROWLOCK
This specifies that a shared row lock be taken when a single shared page or table lock is normally taken.

READUNCOMMITTED | READCOMMITTED |
REPEATABLEREAD | SERIALIZABLE
These hints specify that SQL Server should use the same locking mechanisms as when the transaction isolation level
is set to the level of the same name. However, the hint controls locking for a single table in a single statement, as
opposed to locking of all tables in all statements in a transaction.

NOLOCK
This allows uncommitted, or dirty, reads. Shared locks are not issued by the scan, and the exclusive locks of others
are not honored. This hint is equivalent to READUNCOMMITTED.

READPAST
This specifies that locked rows are skipped (read past). READPAST applies only to transactions operating at READ
COMMITTED isolation and reads past row-level locks only.

XLOCK
This takes an exclusive lock that will be held until the end of the transaction on all data processed by the statement.
This lock can be specified with either PAGLOCK or TABLOCK, in which case the exclusive lock applies to the
appropriate level of granularity.
This document is created with the unregistered version of CHM2PDF Pilot

Setting a Lock Timeout

Although it is not specifically a query hint, SET LOCK_TIMEOUT also lets you control SQL Server locking
behavior. By default, SQL Server does not time out when waiting for a lock; it assumes optimistically that the lock will
be released eventually. Most client programming interfaces allow you to set a general timeout limit for the connection
so that a query is automatically canceled by the client if no response comes back after a specified amount of time.
However, the message that comes back when the time period is exceeded does not indicate the cause of the
cancellation; it could be because of a lock not being released, it could be because of a slow network, or it could just
be a long running query.

Like other SET options, SET LOCK_TIMEOUT is valid only for your current connection. Its value is expressed in
milliseconds and can be accessed by using the system function @@LOCK_TIMEOUT. This example sets the
LOCK_TIMEOUT value to 5 seconds and then retrieves that value for display:
SET LOCK_TIMEOUT 5000
SELECT @@LOCK_TIMEOUT

If your connection exceeds the lock timeout value, you receive the following error message:
Server: Msg 1222, Level 16, State 50, Line 0
Lock request time out period exceeded.

Setting the LOCK_TIMEOUT value to 0 means that SQL Server does not wait at all for locks. It basically cancels
the entire statement and goes on to the next one in the batch. This is not the same as the READPAST hint, which
skips individual rows.

The following example illustrates the difference between READPAST, READUNCOMMITTED, and setting
LOCK_TIMEOUT to 0. All of these techniques let you "get around" locking problems, but the behavior is slightly
different in each case.
1. In a new query window, execute the following batch to lock one row in the titles table:
USE pubs
BEGIN TRANSACTION
UPDATE titles
SET price = price * 10
WHERE title_id = 'BU1032'

2. Open a second connection, and execute the following statements:

USE pubs
SET LOCK_TIMEOUT 0
SELECT * FROM titles
SELECT * FROM authors

Notice that after error 1222 is received, the second select statement is executed, returning all the rows from
the authors table.
3. Open a third connection, and execute the following statements:
USE pubs
SELECT * FROM titles (READPAST)
SELECT * FROM authors

SQL Server skips (reads past) only one row, and the remaining 17 rows of titles are returned, followed by
all the authors rows.
This document is created with the unregistered version of CHM2PDF Pilot

4. Open a fourth connection, and execute the following statements:

USE pubs
SELECT * FROM titles (READUNCOMMITTED)
SELECT * FROM authors

In this case, SQL Server does not skip anything. It reads all 18 rows from titles, but the row for title
BU1032 shows the dirty data that you changed in step 1. This data has not yet been committed and is subject
to being rolled back.
NOTE
The NOLOCK, READUNCOMMITTED, and READPAST table hints are not allowed for tables that are targets
of delete, insert, or update operations.

Combining Hints
You can combine index and lock hints and use different hints for different tables, and you can combine HOLDLOCK
with the level of shared locks. Here's an example:
BEGIN TRAN
SELECT title, au_lname
FROM titles (TABLOCK) JOIN titleauthor
ON titles.title_id=titleauthor.title_id
JOIN authors (PAGLOCK, HOLDLOCK, INDEX(1))
ON authors.au_id=titleauthor.au_id
This document is created with the unregistered version of CHM2PDF Pilot

34

Segregating OLTP and DSS

Applications
Sometimes it makes sense to split up your OLTP and DSS applications. This can be an excellent strategy if your
DSS applications don't need immediate access to information. This is a reason for the recent popularity of data
warehousing, data marts, and other data management systems, although these concepts have been around for years.

You can use a separate database (on the same server or on different servers) for DSS and OLTP, and the DSS
database can be much more heavily indexed than the OLTP database. The DSS database will not have exclusive
locks holding up queries, and the OLTP database's transactions will not get held up by the shared locks of the DSS.
SQL Server's built-in replication capabilities make it relatively easy to publish data from the OLTP server and
subscribe to it from your reporting server. SQL Server's replication capabilities can propagate data in near real time,
with latency between the servers of just a few seconds. However, for maintaining a DSS server, it is best to
propagate the changes during off-peak hours. Otherwise, the locks acquired during propagation at the subscribing site
will affect the DSS users, just as any other update activity would. In addition, if for most of the day the DSS server is
only executing SELECT queries, you can enable the database option read only. This means that SQL Server will not
acquire or check locks for any queries because they will all be shared and therefore compatible. Completely
bypassing lock maintenance can lead to noticeable performance improvements for the SELECT queries.
This document is created with the unregistered version of CHM2PDF Pilot

34

Environmental Concerns
To finish this discussion of query tuning, we'll briefly look at some of the environmental concerns that you need to be
aware of in your programming—for example, we'll discuss case sensitivity, which can greatly affect your applications.
We'll also look at nullability issues and ANSI compatibility.

Case Sensitivity
Various options and settings affect the semantics of your Transact-SQL statements. You must be sure that your
Transact-SQL code can work regardless of the setting, or you must control the environment so that you know what
the setting is.

Case sensitivity is by far the most common environmental problem, and it is simple to avoid. You should seriously
consider doing most of your development in a case-sensitive environment, even if you will deploy your application
mostly in a case-insensitive environment. The reason is simple: nearly all operations that work in a case-sensitive
environment will also work in a case-insensitive environment, but the converse is not true.

For example, if we write the statement select * from authors in the pubs database of a case-sensitive environment,
it will work equally well in a case-insensitive environment. On the other hand, the statement SELECT * FROM
AUTHORS will work fine in a case-insensitive environment but will fail in a case-sensitive environment. The table in
pubs is actually named authors, which is lowercase. The only statement that would work in a case-sensitive
environment but fail in a case-insensitive environment is the declaration of an object name, a column name, or a
variable name. For example, with the statement declare @myvar int, another statement declaring @MYVAR int
would work fine in a case-sensitive environment because the two names are distinct, but it would fail in a
case-insensitive environment because the names would be considered duplicates.

The easiest way to determine whether your environment is case sensitive is to perform a SELECT statement with a
WHERE clause that compares a lowercase letter with its uppercase counterpart—you don't need to access a table to
do this. The following simple SELECT statement returns 1 if the environment is case sensitive and 0 if the environment
is not case sensitive:
SELECT CASE
WHEN ('A'='a') THEN 0
ELSE 1
END

I use the term environment here because in SQL Server 2000 you can control the case sensitivity at many different
levels. Recall from Chapter 4 that your SQL Server instance will have a default collation, which determines both the
character set and the sort order used. Each database can specify the collation it will use in the CREATE DATABASE
command, or you can change the collation using the ALTER DATABSE command. If you don't specify a collation for
a database, it will use the server's default. Case sensitivity is just one of the issues surrounding the character set used
by SQL Server. The character set choice affects both the rows selected and their ordering in the result set for a query
such as this:
SELECT au_lname, au_fname FROM authors
WHERE au_lname='José'
ORDER BY au_lname, au_fname

If you never use characters that are not in the standard ASCII character set, case sensitivity is really your primary
This document is created with the unregistered version of CHM2PDF Pilot

issue. But if your data has special characters such as the é in the example above, be sure that you understand all the
issues surrounding different collations. For more information, see Chapter 4.

Nullability and ANSI Compliance Settings

To be recognized as ANSI compliant, various options had to be enabled in SQL Server because of subtle
differences in semantics between the traditional SQL Server behavior and what is mandated by ANSI. I already
covered the majority of these issues in earlier chapters. To preserve backward compatibility, the prior behavior
couldn't simply be changed. So options were added (or in a few cases, previous options were toggled on) to change
the semantics to comply with the ANSI SQL requirements. These options are summarized in Table 16-3, along with
the statement used to change the behavior.

All of the options listed in the table can be set individually, but you might want to avoid doing that because there are
128 (27) permutations to consider. You might want to set just a few of the options individually, such as SET
ANSI_WARNINGS. But by and large, you should either leave them all at their default settings or change them as a
group to the ANSI SQL-92 behavior. You can enable these options as a group by setting SET ANSI_DEFAULTS
ON. A good reason to use all the ANSI-compliant options is that several of the new features of SQL Server 2000,
such as Indexed Views, require that almost all of these options be set for compliance.

Table 16-3. ANSI SQL requirements satisfied by SQL Server options.

ANSI SQL Requirement SQL Server Statement
Disable SQL Server's = NULL extension SET ANSI_NULLS ON
Automatically display a warning if a truncation would
occur because the target column is too small to accept a
SET ANSI_WARNINGS ON
value. By default, SQL Server truncates without any
warning.
Always right-pad char columns, and don't trim trailing
blanks that were entered in varchar columns, as SQL SET ANSI_PADDING ON
Server would do by default.
Make all statements implicitly part of a transaction,
SET IMPLICIT_TRANSACTIONS ON
requiring a subsequent COMMIT or ROLLBACK.
Close any open cursors upon COMMIT of a transaction.
By default, SQL Server keeps the cursor open so that it
SET CURSOR_CLOSE_ON_COMMIT ON
can be reused without incurring the overhead of reopening
it.
Allow identifier names to include SQL Server keywords if
the identifier is included in double quotation marks, which
SET QUOTED_IDENTIFIER ON
by default is not allowed. This causes single and double
quotation marks to be treated differently.
By default, create as NULL a column in a CREATE
TABLE statement that is not specified as NULL or NOT
NULL. The statement toggles this so that the column can
SET ANSI_NULL_DFLT_ON ON
be created with NULL. (I recommend that you always
specify NULL or NOT NULL so that this setting option
is irrelevant.)

The ability to set these options on a per-connection basis makes life interesting for you as a SQL Server application
programmer. Your challenge is to recognize and deal with the fact that these settings will change the behavior of your
code. Basically, that means you need to adopt some form of the following four strategies:
 The Optimistic Approach Hope that none of your users or the person doing database administration will
change such a setting. Augment your optimism by educating users not to change these settings.
 The Flexible Approach Try to write all your procedures to accommodate all permutations of the settings of
This document is created with the unregistered version of CHM2PDF Pilot

all the various options. (This approach is usually not practical.)

 The Hard-Line Approach Explicitly set your preferences at startup and periodically recheck them to
determine that they have not been subsequently altered. Simply refuse to run if the settings are not exactly
what you expect.
 The Clean Room Approach Have a "sealed" system that prevents anyone from having direct access to
change such a setting.

The approach you take is your choice, but recognize that if you don't think about the issue at all, you have basically
settled for the Optimistic Approach. This approach is certainly adequate for many applications for which it's pretty
clear that the user community would have neither the desire nor the ability to make environmental changes. But if
you're deploying an application and the machine running SQL Server will be accessed by applications that you don't
control, it's probably an overly simplistic approach. Philosophically, the Flexible Approach is nice, but it probably isn't
realistic unless the application is quite simple.

You can change the SQL Server default values for the server as a whole by using sp_configure 'user options'. A
specific user connection can then further refine the environment by issuing one of the specific SET statements just
discussed. The system function @@OPTIONS can then be examined by any connection to see the current settings
for that connection. The @@OPTIONS function and the value to be set using sp_configure 'user options' are a bit
mask with the values depicted in Table 16-4.

By default, none of these options is enabled for SQL Server itself. So in a brand-new installation, the run value for
sp_configure 'user options' will be 0. A system administrator can set this so that all connections have the same initial
default settings. If you query the value of @@OPTIONS from an application that has not modified the environment,
the value will also be 0. However, be aware that many applications, or even the SQL Server ODBC or OLE DB
drivers that the application uses, might have changed the environment. Note that this includes SQL Query Analyzer,
which uses ODBC. To change the default behavior, simply set the corresponding bit by doing a bitwise OR with the
previous value. For example, suppose your run value is 512, which indicates that NOCOUNT is the only option
turned on. You want to leave NOCOUNT enabled, but you also want to enable option value 32, which controls how
NULLs are handled when using equality comparisons. You'd simply pass the decimal value 544 (or 0x220) to
sp_configure 'user options', which is the result of doing a bitwise OR between the two options (for example,
SELECT 32|512).

Table 16-4. Bitmask values for SQL Server options.

Decimal Value Hexadecimal Value Option Description
Controls whether a
IMPLICIT_TRANSACTI transaction is started
2 0x0002
ONS implicitly when a statement is
executed.
Controls behavior of cursors
CURSOR_CLOSE_ON_C
4 0x0004 once a commit has been
OMMIT
performed.
Controls truncation and
8 0x0008 ANSI_WARNINGS NULL in aggregate
warnings.
Controls padding of
16 0x0010 ANSI_PADDING
variables.
Controls NULL handling by
32 0x0020 ANSI_NULLS
using equality operators.
Terminates a query when an
overflow or divide-by-zero
64 0x0040 ARITHABORT
error occurs during query
execution.
This document is created with the unregistered version of CHM2PDF Pilot

Returns NULL when an

128 0x0080 ARITHIGNORE overflow or divide-by-zero
error occurs during a query.
Differentiates between single
and double quotation marks
when evaluating an
expression, allowing object
names to include characters
256 0x0100 QUOTED_IDENTIFIER
that would otherwise not
conform to naming rules or
would collide with a
reserved word or a
keyword.
Turns off the message
returned at the end of each
512 0x0200 NOCOUNT statement that states how
many rows were affected by
the statement.
Alters the session's behavior
to use ANSI compatibility
for nullability. New columns
1024 0x0400 ANSI_NULL_DFLT_ON
defined without explicit
nullability will be defined to
allow NULLs.
Alters the session's behavior
to not use ANSI
compatibility for nullability.
2048 0x0800 ANSI_NULL_DFLT_OFF New columns defined
without explicit nullability will
be defined to prohibit
NULLs.

You can use DBCC USEROPTIONS to examine current options that have been set. The output is similar to the
following code:
Set Option Value
---------- ----------
textsize 64512
language us_english
dateformat mdy
datefirst 7
arithabort SET
nocount SET

This DBCC command shows only options that have been set—it doesn't show all the current settings for
sp_configure 'user options'. But you can also decode your current connection settings pretty easily from
@@OPTIONS using something like this:
SELECT 'IMPLICIT TRANSACTIONS' AS 'OPTION', CASE WHEN
(@@OPTIONS & 0x0002 > 0) THEN 'ON' ELSE 'OFF' END
UNION
SELECT 'CURSOR_CLOSE_ON_COMMIT', CASE WHEN
(@@OPTIONS & 0x0004 > 0) THEN 'ON' ELSE 'OFF' END
UNION
SELECT 'ANSI_WARNINGS',CASE WHEN (@@OPTIONS & 0x0008 > 0) THEN
This document is created with the unregistered version of CHM2PDF Pilot

'ON' ELSE 'OFF' END

UNION
SELECT 'ANSI_PADDINGS', CASE WHEN (@@OPTIONS & 0x0010 > 0) THEN
'ON' ELSE 'OFF' END
UNION
SELECT 'ANSI_NULLS', CASE WHEN (@@OPTIONS & 0x0020 > 0) THEN 'ON'
ELSE 'OFF' END
UNION
SELECT 'ARITHABORT', CASE WHEN (@@OPTIONS & 0x0040 > 0) THEN 'ON'
ELSE 'OFF' END
UNION
SELECT 'ARITHIGNORE', CASE WHEN (@@OPTIONS & 0x0080 > 0)
THEN 'ON' ELSE 'OFF' END
UNION
SELECT 'QUOTED_IDENTIFIER', CASE WHEN (@@OPTIONS & 0x0100 > 0)
THEN 'ON' ELSE 'OFF' END
UNION
SELECT 'NOCOUNT', CASE WHEN (@@OPTIONS & 0x0200 > 0) THEN 'ON'
ELSE 'OFF' END
UNION
SELECT 'ANSI_NULL_DFLT_ON', CASE WHEN (@@OPTIONS & 0x0400 > 0)
THEN 'ON' ELSE 'OFF' END
UNION
SELECT 'ANSI_NULL_DFLT_OFF', CASE WHEN (@@OPTIONS & 0x0800 > 0)
THEN 'ON' ELSE 'OFF' END
ORDER BY 'OPTION'

Here's the result:

OPTION SETTING
---------------------- -------
ANSI_NULL_DFLT_OFF OFF
ANSI_NULL_DFLT_ON ON
ANSI_NULLS ON
ANSI_PADDINGS ON
ANSI_WARNINGS ON
ARITHABORT ON
ARITHIGNORE OFF
CURSOR_CLOSE_ON_COMMIT OFF
IMPLICIT TRANSACTIONS OFF
NOCOUNT OFF
QUOTED_IDENTIFIER ON

Unfortunately, running the previous SELECT statement will not give you a complete picture. As you saw in Chapter
5, many of the options, especially those involving NULL handling, can also be changed at the database level by using
sp_dboption. To really get the full picture, you must examine @@OPTIONS for your current session in addition to
sp_dboption for your current database.

Locale-Specific SET Options

Beware of the locale-specific SET options. SET DATEFORMAT and SET DATEFIRST change the recognized
default date format. If DATEFORMAT is changed to dmy instead of the (U.S.) default mdy, a date such as 12/10/98
will be interpreted as October 12, 1998. A good strategy for dates is to always use the ISO format yyyymmdd,
which is recognized no matter what the setting of DATEFORMAT is.

DATEFIRST affects what is considered the first day of the week. By default (in the United States), it has the value 7
(Sunday). Date functions that work with the day-of-week as a value between 1 and 7 will be affected by this setting.
These day-of-week values can be confusing because their numbering depends on the DATEFIRST value, but the
This document is created with the unregistered version of CHM2PDF Pilot

values for DATEFIRST don't change. For example, as far as DATEFIRST is concerned, Sunday's value is always 7.
But if you've designated Sunday (7) as DATEFIRST, if you execute the statement SELECT
DATEPART(dw,GETDATE()) and your date falls on a Sunday, the statement will return 1. You just defined Sunday
to be the first day of the week, so 1 is correct.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
This chapter walked you through performance-related topics that you should consider when designing and
implementing a high-performance application. As a starting point, skilled team members are vital. More than anything
else, a skilled and experienced staff is the best predictor and indicator of a project's potential success. You must
establish performance criteria and prototype, test, and monitor performance throughout development.

We also looked at techniques that you can use in database design and in making indexing decisions. We discussed
how to deal with lock contention issues and how to resolve deadlocks.

As you make and monitor changes, remember to change just one variable at a time. When you change multiple
variables at the same time, it becomes impossible to evaluate the effectiveness of any one of them. This goes for all
changes, whether they're related to design, indexing, the use of a query optimizer hint, or a configuration setting. This
is a fundamental aspect of all performance optimization. In the next chapter, we'll look at configuration and
performance monitoring.
This document is created with the unregistered version of CHM2PDF Pilot

34

Chapter 17

Configuration and Performance

Monitoring
In the previous chapter, we looked at the most important aspects of performance—those affected by design,
prototyping, indexing, and query writing. You can also improve system performance by tuning your system
configuration, but this is unlikely to yield as much gain as a design change. It's even more critical that your system not
be misconfigured. You might see only a 5 percent improvement in performance by moving from a reasonable
configuration to an ideal configuration, but a badly misconfigured system can kill your application's performance.

This chapter discusses configuration options for Microsoft Windows NT and Windows 2000 and Microsoft SQL
Server 2000. It includes guidelines for setting important values and suggests improvements (which will often be minor
at best). You'll also learn how to use the two most essential monitoring tools: SQL Profiler and System Monitor.
This document is created with the unregistered version of CHM2PDF Pilot

34

Operating System Configuration

Settings
As mentioned above, a badly configured system can destroy performance. For example, a system with an incorrectly
configured memory setting can break an application. SQL Server dynamically adjusts the most important
configuration options for you; you should accept these defaults unless you have a good reason to change them. In
certain cases, tweaking the settings rather than letting SQL Server dynamically adjust them might lead to a tiny
performance improvement, but your time is probably better spent on application and database design, indexing, query
tuning, and other such activities.

Task Management
As you saw in Chapter 3, the operating system schedules all threads in the system for execution. Each thread of
every process has a priority, and Windows NT and Windows 2000 executes the next available thread with the
highest priority. By default, it gives active applications a higher priority. But this priority setting is not appropriate for a
server application that's running in the background, such as SQL Server. To remedy this situation, SQL Server's
installation program eliminates Windows NT and Windows 2000's favoring of foreground applications by modifying
the priority setting. It's not a bad idea to periodically double-check this setting in case someone sets it back. From
Control Panel in Windows NT or Windows 2000, double-click on the System icon to open the System Properties
dialog box. In Windows 2000, you can select the Advanced tab and then click the Performance Options button.
Under Optimize Performance For, be sure that the button for Background Services is selected, and not the one for
Applications. In Windows NT, you can select the Performance tab, and in the Application Performance section, slide
the Select The Performance Boost For The Foreground Application control to the None setting.

Resource Allocation
A computer running Windows NT Server or Windows 2000 Server for file and print services will want to use the
most memory for file caching. But if it is also running SQL Server, it makes sense for more memory to be available for
SQL Server's use. In Windows 2000, you can change the properties for file and print services by right-clicking on
My Network Places and choosing Properties. Then right-click on Local Area Connection and again choose
Properties. Select File And Printer Sharing For Microsoft Networks and click the Properties button. In Windows
NT, you can configure the server's resource allocation (and associated nonpaged memory pool usage) in the Server
dialog box of the Network applet. Double-click on the Network icon in Control Panel and click on the Services tab.
The Server dialog box appears, as shown in Figure 17-1. This dialog box is almost identical to the one you get in
Windows 2000.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 17-1. Configuring the server file and print service settings.

The Maximize Throughput For Network Applications setting is best for running SQL Server. You need high network
throughput with minimal memory devoted to file caching so more memory is available to SQL Server. When this
option is set, network applications such as SQL Server have priority over the file cache for access to memory.
Although you might expect that the Minimize Memory Used option would help, it minimizes the memory available for
some internal network settings that are needed to support a lot of SQL Server users.

PAGEFILE.SYS Location
If possible, you should place the operating system paging file on a different drive than the files used by SQL Server.
This is vital if your system will be paging. However, an even better approach is to add memory or change the SQL
Server memory configuration to effectively eliminate paging. So little page file activity will occur that the file's location
will be irrelevant.

File System Selection

As I discussed in Chapter 4, whether you use the FAT or the NTFS file system doesn't have much effect on system
performance. You should use NTFS in most cases for reasons of security and robustness, not performance.

Nonessential Services
You should disable any services you don't need. In Windows 2000, you can right-click on My Computer and
choose Manage, and in Windows NT, you can use the Services icon in Control Panel. Unnecessary services add
overhead to the system and use resources that could otherwise go to SQL Server. If possible, don't use the server
that's running SQL Server as a domain controller (DC), the group's file or print server, the Web server, the DHCP
server, and so on. You should also consider disabling the Alerter, ClipBook, Computer Browser, Messenger,
Network DDE, and Task Scheduler services that are enabled by default but are not needed by SQL Server.

Network Protocols
You should run only the network protocols you actually need for connectivity. Installing multiple network protocols
This document is created with the unregistered version of CHM2PDF Pilot

(such as TCP/IP, NWLink, NetBEUI, DLC, and Appletalk) on your system is so easy that some systems run
protocols that aren't even used. But this adds overhead. If you legitimately need to use multiple protocols, you should
make sure that the highest binding is set for the most often used protocol.
This document is created with the unregistered version of CHM2PDF Pilot

34

SQL Server Configuration

Settings
Configuration changes can degrade system performance just as easily as they can improve it. This is particularly true
in version 2000 of SQL Server, for which most performance-oriented configuration settings are fully automatic. SQL
Server 2000 has only 10 configuration options that are not considered "advanced," and none of these directly affects
performance. To see all the configuration options, you must change the value of the Show Advanced Options setting:
EXEC sp_configure 'show advanced options', 1
GO

RECONFIGURE
GO

You should change configuration options only when you have a clear reason for doing so, and you should closely
monitor the effects of each change to determine whether the change improved or degraded performance. Always
make and monitor changes one at a time.

Serverwide Options
The serverwide options discussed here are set via the sp_configure system stored procedure. Many of them can
also be set from SQL Server Enterprise Manager, but there is no single dialog box from which all configuration
settings can be seen or changed. Most of the options that can be changed from SQL Server Enterprise Manager user
interface are controlled from one of the Properties tabs that you reach by right-clicking on your server. If you use the
sp_configure stored procedure, no changes take effect until the RECONFIGURE command (or RECONFIGURE
WITH OVERRIDE, in some cases) is issued. Some changes (dynamic changes) take effect immediately upon
reconfiguration, but others do not take effect until the server is restarted. If an option's run_value and config_value
as displayed by sp_configure are different, you must restart the server in order for the config_value to take effect.

We won't look at every configuration option here—only those that relate directly to performance. In most cases,
we'll discuss options that you should not change. Some of these are resource settings that relate to performance only
in that they consume memory (for example, open objects). But if they are configured too high, they can rob a system
of memory and degrade performance. I'll group the sp_configure settings by functionality. Keep in mind that SQL
Server sets almost all of these options automatically, and your applications will work well without you ever looking at
these options.

Memory Options
Before version 7 of SQL Server, you had to manually configure memory options; this is no longer the case. In
Chapter 3, you saw how SQL Server uses memory, including how it allocates memory for different uses and when it
reads data from or writes data to disk. But we did not discuss how much memory SQL Server actually uses for these
purposes or how to monitor the amount of the system's memory resources it uses.

min server memory and max server memory By default, SQL Server automatically adjusts the total amount of the
memory resources it will use. However, you can use the min server memory and max server memory configuration
options to take manual control. The default setting for min server memory is 0 (MB), and the default setting for max
server memory is 2147483647. If you use the sp_configure stored procedure to change both of these options to the
This document is created with the unregistered version of CHM2PDF Pilot

same value, you basically take full control and tell SQL Server to use a fixed memory size. This mimics the memory
configuration behavior prior to SQL Server 7. You can also use SQL Server Enterprise Manager to set a minimum
and maximum value for memory, using the Memory tab of the SQL Server Properties dialog box (shown in Figure
17-2). To bring up the dialog box, right-click on the name of your server and then choose Properties.

Figure 17-2. The Memory tab of the SQL Server Properties dialog box.

You can set the minimum and maximum memory using slider bars. Note that the maximum allowed is slightly less than
the amount of physical memory in the machine. The absolute maximum of 2147483647 MB previously mentioned is
actually the largest value that can be stored in the integer field of the sysconfigures table. It is not related to the actual
resources of your system. You can select a separate option button to indicate that you want to use a fixed size. Setting
the minimum and maximum sliders to the same value has the same effect.

set working set size The Memory tab of the SQL Server Properties dialog box also includes a check box labeled
Reserve Physical Memory For SQL Server. Selecting this is equivalent to setting the configuration option set working
set size to 1, which reserves physical memory space for SQL Server that is equal to the server memory setting. Do
this only if you don't want SQL Server to dynamically adjust its memory usage. Setting set working set size means
that the operating system does not swap out SQL Server pages even if they can be used more readily by another
process when SQL Server is idle. If the operating system's virtual memory manager must page, it must do so from
other processes. These other processes can include other instances of SQL Server. This setting defaults to 0, which
allows the Windows NT or Windows 2000 virtual memory manager to determine the working set size of SQL
Server.

As I've mentioned, you should generally let SQL Server take full control of the memory values, especially if you have
multiple instances running, but if your server might be inactive for long periods, you can manually set min server
memory so that when someone finally submits a query, memory will be available and the query can be processed
quickly. If you have other critical applications on your server that might have inactive periods, you can manually set
max server memory so that SQL Server will not use too much system memory and so that when the other
This document is created with the unregistered version of CHM2PDF Pilot

application becomes active again, memory will be available for it to use.

When SQL Server uses memory dynamically, the lazywriter queries the system periodically to determine the amount
of free physical memory available. The lazywriter expands or shrinks the buffer cache to keep the operating system's
free physical memory at 5 MB plus or minus 200 KB to prevent paging. If less than 5 MB is free, the lazywriter
releases memory to the operating system that usually goes on the free list. If more than 5 MB of physical memory is
free, the lazywriter recommits memory to the buffer cache. The lazywriter recommits memory to the buffer cache only
when it repopulates the free list; a server at rest does not grow its buffer cache.

SQL Server also releases memory to the operating system if it detects that too much paging is taking place. You can
tell when SQL Server increases or decreases its total memory use by using SQL Profiler to monitor the Server
Memory Change event (in the Server category). An event is generated whenever SQL Server's memory increases or
decreases by 1 MB or 5 percent of the maximum server memory, whichever is greater. You can look at the value of
the data element called Event Sub Class to see whether the change was an increase or a decrease. An Event Sub
Class value of 1 means a memory increase; a value of 2 means a memory decrease. We'll look at SQL Profiler in
more detail later in this chapter.

awe enabled A third memory option called awe enabled is available only for SQL Server 2000 Enterprise (or
Developer) Edition running under Windows 2000 Advanced Server or DataCenter Server. This option enables the
use of the Windows 2000 Address Windowing Extensions (AWE) API to support very large memory sizes. SQL
Server 2000 can use as much memory as Windows 2000 Advanced Server or Windows 2000 DataCenter Server
allows. Instances of SQL Server 2000 do not dynamically manage the size of the address space when you enable
AWE memory. Therefore, when you enable AWE memory and start an instance of SQL Server 2000, one of the
following occurs:
 If less than 3 GB of free memory is available on the computer, memory is dynamically allocated and,
regardless of the parameter setting for awe enabled, SQL Server will run in non-AWE mode.
 If sp_configure max server memory has been set and at least 3 GB of free memory is available on the
computer, the instance acquires the amount of memory specified in max server memory. If the amount of
memory available on the computer is less than max server memory (but more than 3 GB) or if max server
memory has not been set, the instance acquires almost all of the available memory and can leave up to only
128 MB of memory free.

If you're running multiple instances of SQL Server 2000 on the same computer and each instance uses AWE
memory, you must pay close attention to certain configuration issues. The following list is taken from SQL Server
2000 Books Online, which documents that you must ensure the following:
 Each instance has a max server memory setting.
 The sum of the max server memory values for all the instances is less than the amount of physical memory in
the computer.

If the sum of the settings exceeds the physical memory on the computer, some of the instances either will not start or
will have less memory than is specified in max server memory. For example, suppose a computer has 16 GB of
physical RAM and has three instances of SQL Server 2000 running on it. Furthermore, max server memory is set to
8 GB for each instance. If you stop and restart all three instances:
 The first instance will start with the full amount of 8 GB of memory.
 The second instance will start, but with slightly less than 8 GB of memory (up to 128 MB less).
 The third instance will start in dynamic memory mode and will have 128 MB or less memory available to it.

user connections SQL Server 2000 dynamically adjusts the number of simultaneous connections to the server if the
user connections configuration setting is left at its default of 0. Even if you set this value to a different number, SQL
Server does not actually allocate the full amount of memory needed for each user connection until a user actually
connects. When SQL Server starts up, it allocates an array of pointers with as many entries as the configured value
for user connections. Each entry uses less than 1000 bytes of memory. Each user who connects to SQL Server is
allocated 24 KB for three memory objects of one page (8 KB) each. These memory objects keep track of things
such as the Process Status Structure (PSS) for each connection and the context area needed by the User Mode
Scheduler (UMS) to keep track of the user's context as it moves the wait queue to a CPU and back again.
This document is created with the unregistered version of CHM2PDF Pilot

You should always let SQL Server dynamically adjust the value of the user connections option. If you configure the
value too low, future connections will be denied access to SQL Server. Even if you're trying to limit the number of
actual simultaneous users, this can backfire because connections are not the same as users. One user can open
multiple connections through the same application. If you're using a query tool such as SQL Query Analyzer, each
time you click the New Query button you start a new connection to the server that counts against the total.

locks The locks configuration option sets the number of available locks (of all types). The default is 0, which means
that SQL Server adjusts this value dynamically. It starts by allocating 2 percent of the memory allotted to SQL Server
to an initial pool of lock structures. When the pool of locks is exhausted, it allocates additional locks. The dynamic
lock pool does not allocate more than 40 percent of the memory allocated to SQL Server. If too much memory is
being used for locks, SQL Server escalates row locks into table locks wherever possible. You should leave this value
at its default. If you manually supply a value for the maximum number of locks, once that number of locks is reached,
any query that requests additional lock resources will receive an error message and the batch will terminate.

Scheduling Options
SQL Server 2000 has a special algorithm for scheduling user processes, which uses UMS threads (as described in
Chapter 3). There is one UMS thread per processor, and the UMS makes sure that only one process can run on a
scheduler at any given time. The UMS manages assignment of user connections to UMS threads to keep the number
of users per CPU as balanced as possible. Four configuration options affect the behavior of the scheduler:
lightweight pooling, affinity mask, priority boost, and max worker threads.

lightweight pooling By default, SQL Server operates in thread mode, which means that the UMS schedules the
operating system threads. However, SQL Server also lets user connections run in fiber mode; fibers are less
expensive to manage than threads. The lightweight pooling option has a value of 0 or 1; 1 means that SQL Server
should run in fiber mode. Using fibers can yield a minor performance advantage, perhaps as much as a 5 percent
increase in throughput when all of the available CPUs are operating at 100 percent. However, the tradeoff is that
certain operations, such as running queries on linked servers or executing extended stored procedures, must run in
thread mode. The cost of switching from fiber to thread mode for those connections can be noticeable, and in some
cases will offset any benefit of operating in fiber mode.

If you're running in an environment with multiple CPUs, all of which are operating at 100 percent capacity, and if
System Monitor shows a lot of context switching, setting lightweight pooling to 1 might yield some performance
benefit. With a system using more than four processors, the performance improvements should be even greater.

affinity mask You can use the affinity mask setting to bind all the threads (or fibers) handled by one UMS thread
to a certain processor. You should avoid doing this because this setting prevents the operating system from using
whatever processor is currently most available; each UMS must then schedule its threads on the same processor.

You can also use this setting to limit the number of processors that SQL Server can use. For example, on an
eight-processor system, setting affinity mask to 63 decimal or 00111111 binary means that SQL Server can use
only processors 0 through 5. Two processors are reserved for other, non_SQL Server activity.

The affinity mask option was added in version 6.5 at the request of some major customers who ran SMP hardware
with more than four processors. These sites were accustomed to similar options on UNIX or mainframe systems.
Most sites do not need this option.

priority boost If the priority boost setting is enabled, SQL Server runs at a higher scheduling priority. The default is
0, which means that SQL Server runs at normal priority whether you're running it on a single-processor machine or on
an SMP machine. Enabling the priority boost option allows the SQL Server process to run at high priority. There are
probably very few sites or applications for which setting this option will make much difference, so I recommend
leaving it alone. But if your machine is totally dedicated to running SQL Server, you might want to enable this option
(set it to 1) to see for yourself. It can potentially offer a performance advantage on a heavily loaded, dedicated
This document is created with the unregistered version of CHM2PDF Pilot

system. Contrary to some folklore, this setting does not make SQL Server run at the highest operating system priority.

max worker threads SQL Server uses the operating system's thread services by keeping a pool of worker threads
(or fibers) that take requests off the queue. It attempts to evenly divide the worker threads among the UMS
schedulers so the number of threads available to each UMS is the setting of max worker threads divided by the
number of CPUs. With 100 or fewer users, there are usually as many worker threads as active users (not just
connected users who are idle). With more users, it often makes sense to have fewer worker threads than active users.
Although some user requests have to wait for a worker thread to become available, total throughput increases
because less context switching occurs.

The max worker threads setting is somewhat autoconfigured. The default is 255, which does not mean that 255
worker threads are in the pool. It means that if a connection is waiting to be serviced and no thread is available, a new
thread is created if the thread total is currently below 255. If this setting is configured to 255 and the highest number
of simultaneously executing commands is, say, 125, the actual number of worker threads will not exceed 125. It might
be even less than that because SQL Server destroys and trims away worker threads that are no longer being used.
You should probably leave this setting alone if your system is handling 100 or fewer simultaneous connections. In that
case, the worker thread pool will not be greater than 100.

Even systems that handle 4000 or more connected users run fine with the default setting of 255. When thousands of
users are simultaneously connected, the actual worker thread pool is usually well below 255 because from the
back-end database perspective most connections are idle, even though the user might be doing plenty of work on the
front end.

Disk I/O Options

No options are available for controlling SQL Server's disk read behavior. All the tuning options to control
read-ahead in previous versions of SQL Server are now handled completely internally. One option is available to
control disk write behavior. This option controls how frequently the checkpoint process writes to disk.

recovery interval The recovery interval option can be automatically configured. SQL Server setup sets it to 0,
which means autoconfiguration. In SQL Server 2000, this means a recovery time of less than one minute, but that
might change in later releases or service packs. This option lets the database administrator control the checkpoint
frequency by specifying the maximum number of minutes that recovery should take.

Prior to version 7 of SQL Server, the recovery interval was set in terms of minutes; the default was 5. In practice,
this meant a checkpoint every 30,000 log records, which was almost always much less than every 5 minutes because
the implementation was not scaled for today's faster systems. In SQL Server 2000, recovery is more likely to match
the recovery interval option, assuming that the system was busy before it came down. For databases with more than
20 MB of log records, the recovery interval directly corresponds to the checkpoint frequency. Thus, a recovery
interval of 5 means that checkpoints occur only every 5 minutes and all dirty pages are written to disk.

Query Processing Options

SQL Server has several options for controlling the resources available for processing queries. As with all the other
tuning options, your best bet is to leave these options at their default values unless thorough testing indicates that a
change might help.

min memory per query When a query requires additional memory resources, the number of pages it gets is
determined partly by the min memory per query option. This option is relevant for sort operations that you
specifically request using an ORDER BY clause, and it also applies to internal memory needed by merge join
operations and by hash join and hash grouping operations This configuration option allows you to specify a minimum
amount of memory (specified in kilobytes) that any of these operations should be granted before they are executed.
Sort, merge and hash operations receive memory in a very dynamic fashion, so you rarely need to adjust this value. In
fact, on larger machines, your sort and hash queries typically get much more than the min memory per query setting,
This document is created with the unregistered version of CHM2PDF Pilot

so you shouldn't restrict yourself unnecessarily. If you need to do a lot of hashing or sorting, however, and you have
few users or a lot of available memory, you might improve performance by adjusting this value.On smaller machines,
setting this value too high can cause virtual memory to page, which hurts server performance.

query wait The query wait option controls how long a query that needs additional memory waits, if that memory is
not available. A setting of -1 means that the query waits 25 times the estimated execution time of the query, but it will
always wait at least 25 seconds with this setting. A value of 0 or more specifies the number of seconds that a query
will wait. If the wait time is exceeded, SQL Server generates error 8645:
Server: Msg 8645, Level 17, State 1, Line 1
A time out occurred while waiting for memory resources to execute the
query. Re-run the query.

Even though memory is allocated dynamically, SQL Server can still run out if the memory resources on the machine
are exhausted. If your queries time out with error 8645, you can try increasing the paging file size or even add more
physical memory. You can also try tuning the query by creating more useful indexes so that hash or merge operations
aren't needed. Keep in mind that this option affects only queries that have to wait for memory needed by hash and
merge operations. Queries that have to wait for other reasons will be unaffected.

index create memory The min memory per query option applies only to sorting and hashing used during query
execution; it does not apply to the sorting that takes place during index creation. Another option, index create
memory, lets you allocate a specific amount of memory for index creation. Its value is also specified in kilobytes.

query governor cost limit You can use the query governor cost limit option to set a maximum length of time that
a query will run. SQL Server will not execute queries that the optimizer estimates will take longer than this. The option
is specified in seconds, but it might not map exactly to seconds on your machine. You should think of the value as an
abstract cost of the query rather than an actual number of seconds. The cost figure was correlated to seconds on one
test machine in the SQL Server development group, and no information is available on the exact specs of that
machine.

You can come up with your own correlation using SQL Profiler. If you capture the Performance: Execution Plan
events, the data element called Binary Data reflects the optimizer's estimated cost for each SQL statement. This value
is compared to the query governor cost limit value. If you then also capture the Duration data for each SQL
statement, you can compare the estimated cost with the actual duration. For example, if, on average, queries that
show a cost (in the SQL Profiler trace) of 8 seconds take 10 seconds to run, you should specify a query governor
cost limit of about 80 percent of what you really want to allow. If you want to block all queries that are estimated to
take longer than 1 minute to run, you should set the option to 48 (80 percent of 60 seconds). We'll look at SQL
Profiler in more detail later in this chapter.

max degree of parallelism and cost threshold for parallelism SQL Server 2000 lets you run certain kinds of
complex queries simultaneously on two or more processors. The queries must lend themselves to being executed in
sections. Here's an example:
SELECT AVG(charge_amt), category
FROM charge
GROUP BY category

If the charge table has 100,000 rows and there are 10 different values for category, SQL Server can split the rows
into groups and have only a subset of the groups processed on each processor. For example, with a 4-CPU machine,
categories 1 through 3 can be averaged on the first processor, categories 4 through 6 on the second processor,
categories 7 and 8 on the third, and categories 9 and 10 on the fourth. Each processor can come up with averages for
only its groups, and the separate averages are brought together for the final result.

During optimization, the optimizer always finds the cheapest possible serial plan before considering parallelism. If this
This document is created with the unregistered version of CHM2PDF Pilot

serial plan costs less than the configured value for the cost threshold for parallelism option, no parallel plan is
generated. The cost threshold for parallelism option refers to the cost of the query in seconds, but just like the
query governor cost limit option, the value is not in actual clock seconds. The default value for the cost threshold
for parallelism option is 5. Otherwise, if the cheapest serial plan costs more than this configured threshold, a parallel
plan is produced based on some assumptions about how many processors and how much memory will actually be
available at run time. This parallel plan cost is compared with the serial plan cost and the cheaper one is chosen. The
other plan is discarded.

When the parallel plan is created, the optimizer inserts exchange operators into the query execution plan to prepare
the query for parallel execution. An exchange operator is an operator in a query execution plan that provides process
management, data redistribution, and flow control. The two main types of exchange operators that you'll observe if
parallel plans are generated are the distribute operator, which separates the processing into multiple streams, and the
gather operator, which retrieves the separate results and combines them into a single result. Occasionally, you'll also
see a redistribute operator if the original streams need to be moved onto different processors at some point during
the processing.

A parallel query execution plan can use more than one thread; a serial execution plan, which is used by a nonparallel
query, uses only a single thread. The actual number of threads used by a parallel query is determined at query plan
execution initialization and is called the degree of parallelism (DOP). This is the algorithm for determining what DOP
to run at, based on the state of the system:
1. The maximum DOP of a query is the minimum of the configured max degree of parallelism value, the
OPTION query hint MAXDOP, if any was specified, and the number of schedulers in the system:
MaxDOP = min ('max degree of parallelism',
'OPTION MAXDOP', #schedulers)

2. The maximum DOP is further limited by dividing half the number of free threads available at the time the
query starts by the number of concurrent "branches" the query might run. Each branch represents one stage of
the query. For example, a branch might be a parallel scan or a parallel hash join. Note, however, that because
SQL Server does not reserve these threads in any way, it might still run short of threads at runtime and get an
8642 error. If this happens, you should consider increasing the max worker threads configuration
parameter:
MaxDOP = min (MaxDOP, 0.5 * #free threads
/ #concurrent branches in this statement)

3. Next, SQL Server determines whether the query is a "large" query. If so, the query will run at the MaxDOP
determined in the previous steps. In SQL Server 2000, a query is considered "large" if the serial cost of the
query is greater than 600 (10 minutes). This limit was added to avoid situations in which a long-running query
starts running at a low DOP because the system is busy and continues to run at that low DOP even if the
system becomes less busy and more resources become available. (See the next step for more details.)
If (optimizer's parallel cost * optimizer's estimated DOP > 600)
Then DOP = MaxDOP

4. If the query is not a large query, SQL Server looks at how busy the system is to decide its DOP. First, it
looks at how many active threads are in the system. A thread is determined to be active if it is being used by
some connection, so this does not include internal system tasks. If there are more than four active threads per
scheduler, SQL Server is very conservative with DOP and runs with DOP 0, which is in effect a serial plan:
If (#active threads > 4 * #schedulers) Then DOP = 0

5. Otherwise, if there are no more than four active threads per scheduler, SQL Server looks at how many
schedulers are currently idle (have no worker threads that are running or capable of running at that instant).
This value is then used as the DOP:
DOP = #idle schedulers
This document is created with the unregistered version of CHM2PDF Pilot

Once a query starts executing on multiple threads for parallel execution, it uses the same number of threads until
completion. Each time a parallel query execution plan is retrieved from the cache, SQL Server reexamines the optimal
number of threads specified. One execution of a query can result in the use of a single thread, and another execution
of the same query (at a different time) can result in using two or more threads.

As you can see in the calculations above, the max degree of parallelism option indicates the maximum number of
processors that are used to run queries in parallel. The default value of 0 means that SQL Server should use the
number of available CPUs. A value of 1 means that SQL Server should suppress parallelism completely. Any other
value restricts the degree of parallelism below the number of CPUs. If your computer has only one processor, the
max degree of parallelism value is ignored.

Be careful when you use the max degree of parallelism or cost threshold for parallelism options—they have
serverwide impact. You can also use trace flag 8687 to turn off parallelism for individual connections.

You can use SQL Profiler to trace the degree of parallelism for each query captured. If you choose to monitor events
in the Performance category, you'll see four different events for Degree Of Parallelism. This is so you can use the SQL
Server 2000 Profiler to trace a SQL Server 7 Server. In that case, a different one of the options is used for each
different DML statement (Delete, Insert, Select, or Update). In SQL Server 2000, tracing any of these four events
will write a row to the trace output for every query indicating its degree of parallelism. The Event Sub Class data is a
number indicating which DML statement was run: 1 = SELECT, 2 = INSERT, 3 = UPDATE, or 4 = DELETE. The
value in the Binary Data column indicates the degree of parallelism actually used for that query—that is, the number of
CPUs used to perform the operation.

Database Options
Several database options can affect performance. The read only and single user options affect performance
because they eliminate locking in some cases. If your database is used only for decision support (for example, a data
mart), it might make sense to set your database to read only, as follows:
ALTER DATABASE 'dbname' SET READ_ONLY

In this way, no locking is performed in the database; this greatly reduces overhead. This option prevents changes
from being made to the database. You can easily toggle it off to perform tasks such as bulk loading of data. For an
operation in which the data is changed via large batch jobs that can run during off-hours and the normal workload is
query only, read only is a good choice.

Even if your database is not read only, you might want to perform off-hour maintenance tasks and bulk loading
operations with the single user option enabled, as shown here:
ALTER DATABASE 'dbname' SET SINGLE_USER

This option also eliminates the need for locking because only one connection can use the database at a time. Of
course, you need locking to make a multiuser system behave like a single-user system. But in this case, it is a
single-user system, so locking is not required. If you need to do a big bulk load and index creation at night while no
one else is using the system, you can eliminate the need to take locks during the bulk load.

Two other options are particularly useful if you're running SQL Server Desktop Edition on a small machine and your
resources are extremely limited: autoclose and autoshrink. The autoclose option causes the database to shut down
cleanly and free resources after the last user exits. By default, it is set to TRUE for all databases when SQL Server
runs on Windows 98. The database reopens automatically when a user tries to use the database. The autoshrink
option (discussed in Chapter 5) can be useful if your disk space resources are extremely limited, but there is a
This document is created with the unregistered version of CHM2PDF Pilot

performance tradeoff. Shrinking a database is a CPU-intensive operation and takes a long time. All indexes on heaps
affected by the shrink must be adjusted because the row locators change.

Finally, two database options control the automatic gathering of statistics: auto create statistics and auto update
statistics. These were both discussed in Chapter 15.

Buffer Manager Options

In Chapter 3, we saw how the buffer manager (also known as the cache manager) uses a queue of "favored pages,"
which you can think of as a least recently used (LRU) algorithm, but no queue list of buffers is maintained in the order
of most recent access. You can use the pintable option to directly influence the favoring behavior.

Pinning a Table in the Cache

You can permanently remove a table from the queue of pages examined by the lazywriter so that once they are read
into the data cache, they are never forced from cache. (This favors them permanently, although it is really more than
that: they are entirely exempted.) You can enable this option using the sp_tableoption stored procedure with the
pintable option.

The pintable option is not appropriate for most sites. But if you get very random cache hits and you have some
relatively small tables that are hot compared to other larger tables, this option might be beneficial. If those small, hot
tables keep getting forced out by the random access to larger tables and you have plenty of memory, you can pin
those small tables. By pinning, you override the buffer manager. But you take away the pages in cache used for that
table, so you give the buffer manager less memory to work with. Pinning the table does not initially read it in; pinning
just makes the table "sticky" so that once a page of the table is in the cache, it doesn't get forced out. If you want the
table preloaded and sticky, you can enable the option and then do SELECT * FROM table to read all the pages into
cache. Be careful when you use this option; the table shouldn't be so big that you eat up more memory than your
system can afford to use.

Monitoring Buffer Manager Performance

Two DBCC commands exist to monitor the performance of the buffer manager; they are known as
SQLPERF(WAITSTATS) and SQLPERF(LRUSTATS). DBCC SQLPERF(WAITSTATS) not only provides an
overview of the buffer manager, it helps identify where a transaction is delayed. It shows the total milliseconds of wait
time and how much of the time is due to waiting for reads, waiting for different types of locks, waiting to add a log
record to the log buffer, and waiting for log buffers to be physically written. The command DBCC
SQLPERF(LRUSTATS) gives details on the lazywriter and the state of the cache, such as the cache-hit ratio, the size
of the cache, the average number of pages scanned to find an available buffer, and the number of free pages. Although
this information is available through System Monitor, the DBCC SQLPERF command lets you capture the output to a
table and save the data for analysis.

For example, to capture the DBCC SQLPERF(WAITSTATS) output, you can create a table with four columns and
use the dynamic EXECUTE command to populate the table. The following script shows one way to define and
populate such a table. It actually adds a fifth column to the table, which indicates the current date and time that the
rows were inserted into the table.
CREATE TABLE waitstats
(
wait_type varchar(20),
requests numeric(10, 0),
wait_time numeric(12, 2),
signal_wait numeric(12, 0),
recorded_time datetime DEFAULT GETDATE()
)
GO
This document is created with the unregistered version of CHM2PDF Pilot

INSERT INTO waitstats (wait_type, requests, wait_time, signal_wait)

EXEC ('DBCC SQLPERF(WAITSTATS)')

Startup Parameters on SQLSERVR.EXE

You can alter startup parameters used by the SQL Server process to tune performance by disabling the performance
statistics collection or specifying trace flags.

Disabling the Performance Collection

Normally, SQL Server keeps performance statistics such as CPU usage and amount of I/O on a per-user basis. The
values are materialized in the sysprocesses table when queried. If you never query or monitor these values, the work
to keep track of them is unnecessary. You can eliminate the calls that produce these performance statistics by passing
the -x startup flag to sqlservr.exe. Or you can add -x as an additional parameter from the Setup program via the Set
Server Options dialog box.

Specifying Trace Flags

We've discussed the use of trace flags several times, and you've seen how to enable them using DBCC TRACEON.
You can also specify a trace flag at server startup; some trace flags make the most sense if they are specified
serverwide rather than on a per-user basis. For example, trace flag 1204, which records information to the error log
when a deadlock occurs, doesn't make sense for a specific connection because you probably have no idea ahead of
time which connections will encounter deadlocks. To enable a trace flag serverwide, you add the flag as a parameter
when you start SQL Server from the command line (for example, sqlservr.exe -c -t1204).
This document is created with the unregistered version of CHM2PDF Pilot

34

System Maintenance
SQL Server requires much less regular system maintenance than most comparable products. For example, because
pages are automatically split, it maintains data clustering characteristics; most products that support data clustering
make you do data reorganization to keep the data clustered. However, you should perform a couple of regular
maintenance tasks for performance reasons. You can easily schedule these tasks (such as updating statistics and
rebuilding your clustered index) using the SQL Server Agent service or the Database Maintenance Plan Wizard.

In the earlier discussion of tuning queries, I explained the importance of index distribution statistics. If you have not
enabled automatic updating of statistics, you should update statistics frequently so that any significant changes in the
volumes or distribution of data are reflected. Although it's not essential to dump and reload tables to keep clustering
properties intact, doing so can be useful in rebuilding tables to reestablish fill factors and avoid page splits. The simple
way to do this is to rebuild the clustered index on the table using the DROP_EXISTING option. When you use this
option, the existing nonclustered indexes do not have to be re-created because the clustering key will stay the same.
You can also use the DBCC DBREINDEX command to rebuild all the indexes on a table. Rebuilding the clustered
index helps keep a table from becoming fragmented and makes read ahead more effective. You can use DBCC
SHOWCONTIG to determine how much fragmentation a table has. To remove fragmentation without incurring the
overhead of completely rebuilding indexes, you can use DBCC INDEXDEFRAG. Of course, you should also
regularly do other tasks, such as performing backups and periodically checking the structural integrity of the database
(for example, by using DBCC CHECKDB), but these are not performance-related tasks.
This document is created with the unregistered version of CHM2PDF Pilot

34

Monitoring System Behavior

Throughout this book, we've looked at statements that are useful for monitoring some aspect of SQL Server's
performance. These include procedures such as sp_who2, sp_lock, SET SHOWPLAN_TEXT ON, DBCC
SQLPERF, and various trace flags for analyzing deadlock issues. SQL Server Enterprise Manager provides a
graphical display that is a combination of sp_who2 and sp_lock, and you should make use of it. In addition, two
other tools let you monitor the behavior of your SQL Server system: SQL Profiler and System Monitor (Performance
Monitor in NT).

SQL Profiler
SQL Profiler has been completely rewritten for SQL Server 2000 because of the need to absolutely guarantee that
no trace records will be lost under stress. SQL Server 2000 must support C2 auditing, and losing any trace
information is unacceptable if auditing is enabled. In addition, if records are missing, it is much more difficult for
Product Support to use SQL Profiler for reproducing problems.

History of SQL Profiler

The original tracing tool, which appeared in version 6.5, was called SQL Trace and was basically an "ODS Sniffer."
SQL Trace was itself based on an earlier freeware product called SQLEye. SQLEye was essentially a filter that
listened on the network using the ODS API for all the batches that were sent to a specified SQL Server. When
SQLEye received an incoming batch, it could apply one of several filters to it and record in a file all batches that
satisfied the filters' conditions. In version 6.5, the same developer who wrote SQLEye put a GUI wrapper around its
configuration and specification commands and created the SQL Trace tool.

Because SQL Trace worked only at the ODS layer, it could capture only the commands that were actually being
sent to SQL Server. Once SQL Server received the command, SQL Trace had no further knowledge of what SQL
Server did with it. For example, if a client connection sent a batch to SQL Server that consisted of only a stored
procedure call, that is all SQL Trace could capture and record. It could not record all the statements that were
executed within the stored procedure, and it had no knowledge of other procedures that were called by the original
stored procedure.

SQL Profiler in versions 7 and 2000 belongs to a new generation of profiling tools. Not only can it keep track of
every statement executed and every statement within every stored procedure, but it can keep track of every time a
table is accessed, every time a lock is acquired, and every time an error occurs.

Tracing in SQL Server 2000

SQL Server 2000 divides the tracing functionality into two separate but connected components. On the server side is
the SQL Trace facility, which manages queues of events generated by event producers. A consumer thread reads
events off of the queues and filters them before sending them to the process that requested them. Events are the main
unit of activity as far as tracing is concerned, and an event can be anything that happens inside SQL Server, or
between SQL Server and a client. For example, opening a table to access its data is an event, executing a stored
procedure is an event, and sending a T-SQL batch from a client to the SQL Server is an event. There is a set of
system store procedures that you can use to define which events should be traced, what data for each event you're
interested in, and where you want to save the information collected from the events. You can specify filters to apply to
the events in order to reduce the amount of information collected and stored. We'll see some examples of the use of
these procedures later in this section.
This document is created with the unregistered version of CHM2PDF Pilot

Figure 17-3 shows the general architecture of the SQL Trace server component. (Event producers are server
components such as the lock manager, buffer manager, and ODS, which generate the events recorded in the queue.)

Figure 17-3. The SQL Trace architecture.

As an alternative to using the system procedures to define the events to trace, you can use the client-side component
called SQL Profiler. Using this tool, you can select the events and data columns from a list and then specify how you
want to filter the events. The tool then displays every event that meets your criteria as it occurs.

You incur a lot more overhead when you use SQL Profiler to capture and save events. The server-side consumer
thread returns OLE-DB row sets to the client-side consumer, and then SQL Profiler must display the data from the
row sets appropriately. To reduce the amount of data sent from the server to the client, each row contains only a
minimum amount of information—basically an event code, a data column code, and a value for that data column. For
example, if through SQL Profiler we have asked to keep track of the text of every single stored procedure when it
completes and we want to know what time it completes, how much time it took to execute the procedure, the name
of the procedure, and the login ID of the user who executed it, the server trace will send multiple rows to the client
every time a stored procedure completes. Each row will contain the event code (43 for Stored Procedure Completed
events) and one of the requested data column codes. There will be one row for the text (the name) of the procedure
call, one row for the end time, one row for the user's login name, and so forth. There will be other rows for data
columns we didn't explicitly request because a set of default data columns is returned with every trace. SQL Profiler
must read these rows and pivot them to produce one row for each event for display purposes.

When you configure the client-side trace using SQL Profiler, you can choose to save the captured data to a file or a
table, in addition to displaying it in the Profiler UI. Once you save trace data, you can again use SQL Profiler to read
the saved data for display or analysis purposes. Figure 17-4 shows the client-side architecture. The controller can
either get the data from the consumer, which received currently active trace information from the server, or it can read
data from a saved location, such as a table or a file. On the left, you can see all the different data sources that can be
read. SQL Server 2000's Profiler can read trace data saved with the SQL Server 7 Profiler, and it can read script
files saved using the trace facility in SQL Server 6.5.

Figure 17-4. The SQL Profiler architecture.

This document is created with the unregistered version of CHM2PDF Pilot

SQL Server 2000 provides more than 100 predefined events, and you can configure 10 additional events of your
own. SQL Profiler is more than just the user interface that is available through the SQL Profiler icon. You use the
interface for defining traces from a client machine, watching the events recorded by those client side traces, and
replaying traces. SQL Profiler can also define server-side traces that run invisibly, using extended stored procedures.
One of the advantages of server-side traces is that they can be configured to start automatically. We'll see more on
server-side traces later.

Defining a Trace
We'll look at how to define a trace by using the SQL Profiler interface, but the details regarding the types of events,
data elements available, and filters are equally applicable to server-side traces defined using the system stored
procedures. To start a new trace, choose New and then Trace from the File menu in SQL Profiler. You'll be asked to
log in to a particular SQL Server, and then you'll see a dialog box like the one shown in Figure 17-5, with four tabs
for defining the trace properties. Alternatively, you can choose File/Open/Trace Template, and then select from 8
predefined trace definitions. You'll get the same tabbed dialog box, but much of the information will be predefined.
From the File menu, you can also open a saved trace file or a trace table that you saved previously.

Figure 17-5. The Trace Properties dialog box.

General Tab

When you define a new trace, you give the trace a name and select the SQL Server you want to monitor. The default
is the server that you just logged on to, but you can select another SQL Server. You can choose to base your new
trace on one of the existing templates by specifying either the name of a template or the file location where the
template is stored. Optionally, you can save the information to a file while it is being captured. The file can be
processed from either the client or the server. If it is processed from the client (SQL Profiler), each row is processed
individually. Writing to a server-side file is done using the same mechanism as log writes use, and the file is opened for
pure sequential writing. You can set a maximum file size in megabytes, and you can configure the trace for automatic
file rollover. This means that when the file reaches the maximum size you specified, SQL Server will automatically start
a new file with a numeric suffix added to the name. This allows the trace files to stay smaller and be managed more
easily. Finally, you can set the trace to stop at a specified time; it will start as soon as you click the Run button.
This document is created with the unregistered version of CHM2PDF Pilot

If you use the SQL Profiler interface to create a server-side file, two traces are actually created—one on the server
to write to the file and one on the client for display purposes. This does not mean that the events are raised twice;
they're raised only once, and the queue filter will manage the fact that there are two subscribers for the same
information.

Events Tab

On the Events tab, you can select from over 100 available events, which are grouped into 13 categories. You can
select an entire category or select events from within a category. The online documentation gives a complete
description of each event. The Events tab is shown in Figure 17-6.

Figure 17-6. The Events tab of the Trace Properties dialog box.

Data Columns Tab

You can specify which data items you want to record for each event. For example, you might want to keep track of
only the SQL Server user, the application being used, and the text of the SQL statement being executed. Or you
might want to record performance information such as the duration of the event, the table or index accessed, the
severity of an error, and the start and end time of the event. Three data elements—BinaryData, IntegerData, and
EventSubClass—have different meanings depending on what event is being captured. For example, if you're capturing
the Execution Plan event, BinaryData reflects the estimated cost of the query and IntegerData reflects the estimate of
the number of rows to be returned. If you're capturing the Server Memory event, EventSubClass data can be 1,
which means that the server memory was increased, or it can be 2, which means it was decreased. The IntegerData
column reflects the new memory size in megabytes. Not all events have meaningful values for every possible data
element. Again, details are available in the online documentation, but they might be a little hard to find. Figure 17-7
shows you where in SQL Server Books Online you can find the details. The figure shows the section on Monitoring
Server Performance And Activity, which is in the Administering SQL Server volume. Under Monitoring With SQL
Profiler Event Categories is a section for each different category. The figure shows the Performance Event Category,
and the right pane shows the top part of the article on Performance Data Columns. The table displays the meaning of
the data columns for each possible event. For the Degree Of Parallelism1 Event Class, the EventSubClass column
This document is created with the unregistered version of CHM2PDF Pilot

indicates the type of operation being performed, the IntegerData column indicates the pages used in memory for the
query plan, and the BinaryData column indicates the actual DOP.

Figure 17-7. Finding the details of each of the data columns in Books Online.

It might seem strange that there are four different events for degree of parallelism, but this is necessary for backward
compatibility, so that SQL Profiler can be used to trace activity on a SQL Server 7 server. If you are tracing events
on a SQL Server 2000 server, all four events will provide the same functionality (for example, any of the four can be
used to get the same DOP information). If you are tracing a SQL Server 7 server, each DOP event will trace only
one kind of DML statement: Degree Of Parallelism1 traces INSERT statements, Degree Of Parallelism2 traces
UDPATE statements, Degree Of Parallelism3 traces DELETE statements, and Degree Of Parallelism4 traces
SELECT statements. The Data Columns tab is shown in Figure 17-8.

Figure 17-8. The Data Columns tab of the Trace Properties dialog box.
NOTE
This document is created with the unregistered version of CHM2PDF Pilot

The companion CD includes a spreadsheet called SQL Trace Event Summary.xls that lists each possible event, its
event ID, and the data columns available for that event.

Filters Tab

Using the Filters tab, you can filter out events that aren't of interest. Two types of filters are available—string filters
and numeric filters. Each data element can be filtered in only one of these ways. For data values that are strings, such
as ApplicationName or DatabaseName, you can apply a pattern matching string using either LIKE or NOT LIKE,
and wildcards are allowed. For example, if I want to trace events only from databases that have history in their name,
I can supply a filter LIKE '%history%'. Of course, I also have to capture the DatabaseName column in order to have
this filter available. For numeric values such as DatabaseID or Duration (in milliseconds), you can apply filters that
check for the data being equal to, not equal to, less than or equal to, or greater than or equal to. For example, if I
want to trace only events that took longer than 1000 milliseconds to execute, I can set the Duration filter to 'greater
than or equal' 1000. For data that represents datetime values, only the greater than and less than filters are available.

By default, all traces exclude events originating from SQL Profiler because typically you don't want to trace the
events that are involved in the actual tracing. Figure 17-9 shows the Filters tab of the Trace Properties dialog box.

Figure 17-9. The Filters tab of the Trace Properties dialog box.

The online documentation provides full details of all the filters and describes which data elements allow which filters.
Note that you can supply multiple values for a condition. For example, if you want to trace events that occur only in
databases with ID 5 or 6, you can enter both 5 and 6 under the Equals condition.

The Filters tab includes a check box labeled Exclude System IDs. Checking this allows you to capture fewer trace
events by excluding most of the events that reference system objects. Selecting this option sets the ObjectID filter to
greater than or equal to 100, thus eliminating most of the system objects from the trace, and possibly reducing the
number of events captured by up to 25 percent.

As a security mechanism, SQL Profiler automatically omits from the trace any of the security-related stored
procedures affecting passwords, and this behavior is always in effect and nonconfigurable. This includes procedures
This document is created with the unregistered version of CHM2PDF Pilot

such as sp_addapprole, sp_password, sp_addlogin, and sp_addlinkedsrvlogin, as well as many of the procedures
for configuring replication.

You can save and restart traces that you define using SQL Profiler. You can also copy the definition to another
machine and start it from the SQL Profiler user interface there.

Information gathered by SQL Profiler or SQL Trace can be extremely useful for tracking all kinds of performance
problems. By saving the captured events to a table, you can analyze usage patterns and query behavior using
Transact-SQL stored procedures. By saving your captured events to a file, you can copy that file to another machine
or e-mail it to a central location (or even to your support provider).

Defining a Server-Side Trace

If you want to start a particular trace programmatically (or automatically), you must define a server-side trace using
the sp_trace_* internal stored procedures. You can include the calls to these procedures in any SQL Server stored
procedure or batch. To define a server-side trace, client programs such as SQL Query Analyzer have to know only
four procedures.
1. sp_trace_create sets up the trace and configures a file to keep the captured events. It returns a trace ID
number that is then used in the other three procedures to further define and manipulate the trace.
2. sp_trace_setevent is then called once for each data column of every event that you want to capture.
3. sp_trace_setfilter can be called once for each filter you want to define on any of the events' data columns.
4. sp_trace_setstatus is called to start, stop or remove the trace.

A fifth procedure, sp_trace_generateevent, can optionally be called to generate user-defined events.

The trickiest part of setting up a server-side trace is that you must use ID numbers when specifying the events and
data columns. Fortunately, the page in the SQL Server Books Online that documents the sp_trace_setevent
procedure provides the ID number for each event and each data column. Here's an example that creates a trace that
writes events to a file called C:\TraceSELECT.trc. It will capture the start time, text, and database ID of each
SQL:BatchCompleted event, but the filter restricts the trace to only capturing events that have the substring 'SELECT'
in the text data column.
DECLARE @RC int, @TraceID int, @ON bit
SET @ON = 1
EXEC @RC = sp_trace_create @TraceID OUTPUT, 0, N'C:\TraceSELECT.trc'
SELECT 'RC'= @RC, 'TraceID'= @TraceID

-- SQL:BatchCompleted Event, Start Time column

EXEC @RC = sp_trace_setevent @TraceID, 12, 14, @ON
-- SQL:BatchCompleted Event, Database ID Column
EXEC @RC = sp_trace_setevent @TraceID, 12, 3, @ON
-- SQL:BatchCompleted Event, Text Column
EXEC @RC = sp_trace_setevent @TraceID, 12, 1, @ON
-- Audit Login event, Start Column
EXEC @RC = sp_trace_setevent @TraceID, 14, 14, @ON

-- Set filter for Column 1 (text), data = 'select'

EXEC @RC = sp_trace_setfilter @TraceID, 1, 0, 0, N'select'

-- Start Trace (status 1 = start)

EXEC @RC = sp_trace_setstatus @TraceID, 1

One crucial aspect of these stored procedures is that their parameters must be passed with the correct datatype. The
filename parameter for sp_trace_create is of type nvarchar, so a Unicode constant with the N prefix must be
supplied for the filename. The last parameter of the sp_trace_setevent procedure, which sets the tracing of that data
for that event to either ON or OFF, must be of type bit. Because I can't do a CAST or CONVERT in the procedure
call to force a constant of 1 to be a bit, I defined a bit variable to hold the value, and I pass that variable to every call.
This document is created with the unregistered version of CHM2PDF Pilot

To modify this trace at a later time, I would need to pass a bit value of 0 to disable the capturing of a particular data
column.

Let's look at the procedure that creates the trace in a bit more detail. Here's the syntax:

sp_trace_create [ @traceid = ] trace_id OUTPUT

, [ @options = ] option_value
, [ @tracefile = ] 'trace_file'
[ , [ @maxfilesize = ] max_file_size ]
[ , [ @stoptime = ] 'stop_time' ]

The @traceid parameter returns an integer ID that is used to modify a trace, stop or restart it, or look at its
properties. The preceding script printed out the value of the trace ID just for informational purposes. The @options
parameter allows us to specify one or more options for this trace. Here are the possible values:
0 No options set.
1 Trace should produce a rowset.
2 Start a new trace file when the current file reaches its maximum size. If no
maximum size has been specified, the default is 5 MB. This is equivalent to
using the Enable File Rollover option in the General tab of the Trace
Properties dialog box.
4 SQL Server should shut down if the file cannot be written to. This is needed
for servers that are being audited.
8 Produce a Blackbox.trc file, which is a record of the last 5 MB of trace
information gathered by the server; cannot be used with any other options.

A value of 1 means that the trace will produce a rowset that will be sent to the SQL Profiler UI, and cannot be used
if we are capturing to a server-side file.

You might find it useful to have certain traces running constantly, but be careful about the amount of information you
capture. Trace files can grow quite large if they are left running the entire time SQL Server is running. To configure a
trace to start automatically every time SQL Server starts, you can put the trace definition into a stored procedure, and
then use sp_procoption to set that procedure's startup property to ON.

SQL Server 2000 provides several functions to inspect the status of your traces. The most useful is probably
fn_trace_getinfo, which reports all the parameters specified in the sp_trace_create procedure, as well as the status
of the trace (1 = currently running). Here's an example of its use:
SELECT * FROM :: fn_trace_getinfo(default)

We could supply a particular trace ID instead of the word "default," but if we don't know the trace ID, using this form
shows all the traces. Here's my output:
traceid property value
----------- ----------- -------------------
1 1 0
1 2 C:\TraceSELECT.trc
1 3 5
1 4 NULL
1 5 1

The row with property 1 contains the value of the @options parameter, as shown in the preceding chart. If you see a
trace with a value of 1 for property 1, it is most likely a trace that was started from the SQL Profiler UI. The row with
property 2 contains the trace filename, if any. Note that SQL Server will automatically append the trc suffix to your
filename, even if you already supplied that suffix. The row with property 3 contains the maximum file size, which
This document is created with the unregistered version of CHM2PDF Pilot

defaults to 5 MB, and the row with property 4 contains the stop time, which has no value for my trace. Finally, the
row with property 5 is the trace status; a 0 means the trace is stopped, a 1 means the trace is running.

The fn_trace_geteventinfo function will show you what events and data columns you are capturing for a particular
trace. The data is returned with the event and data column IDs, so you'll have to do a little work to translate. The
function fn_trace_getfilterinfo will return information about the filters on a particular trace.

You might have noticed that there is nothing in the supplied procedures to allow capturing trace data to a table for
traces that you define on the server. This is by design, because writing to a table cannot be done as quickly as writing
to a file, and it is imperative that the trace capturing operations not lose any events, no matter how busy your server is.
If you want to view your trace data in a table, so that you can analyze it using SQL queries, you can copy the data
from a trace file to a table using the function fn_trace_gettable.

Server-side traces are wonderfully useful in all kinds of situations and much more efficient that client side tracing with
SQL Profiler. If you are avoiding them because you think it's too hard to write all the procedure calls and look up all
the necessary event and data column ID information, take heart. You can use the SQL Profiler UI to define all the
events, all the data columns and all the filters that you want, and then use the item in the File menu called Script Trace,
which will produce all the calls to the stored procedures that correspond to the trace elements you select. The only
change you'll need to make is to provide a filename to hold the captured data, and then you can run the newly created
script through SQL Query Analyzer to start your trace running as a server-side trace.

Gathering Diagnostic Information

A lot of the information that your traces gather can help your support provider troubleshoot system problems.
However, this information is usually not enough by itself. Support providers ask a standard set of questions when you
call. A special utility in SQL Server 2000 helps you gather this information with a single command: sqldiag. It can
work with a SQL Server trace if you have enabled automatic query recording using the blackbox functionality. I'll talk
about this blackbox later in this section. When you execute sqldiag from a command prompt while SQL Server is
running, it gathers the following information:
 Text of all error logs (unless the -X, upper case, option has been supplied)
 Registry information
 DLL version information
 Output from:
 sp_configure
 sp_who
 sp_lock
 sp_helpdb
 xp_msver
 sp_helpextendedproc
 sysprocesses
 Input buffer SPIDs/deadlock information
 Microsoft Diagnostics Report for the server, including:
 Contents of <servername>.txt file
 Operating System Version Report
 System Report
 Processor List
 Video Display Report
 Hard Drive Report
 Memory Report
 Services Report
 Drivers Report
 IRQ and Port Report
 DMA and Memory Report
 Environment Report
This document is created with the unregistered version of CHM2PDF Pilot

 Network Report

The information is stored in a file with the default name of sqldiag.txt; you can specify a different filename in the
command line for sqldiag. The default directory is the \LOG directory of your SQL Server installation. Obviously, if
SQL Server is not running, some of the information in the preceding list won't be available, but any blackbox
information will be captured.

To enable blackbox recording, create and start a trace with an option value of 8; when executing the
sp_trace_create procedure, don't specify a filename or any other parameters. A trace file called blackbox.trc will
automatically be captured in the default \Data directory of your SQL Server installation. If you look at your trace
properties with fn_trace_getinfo you can see the exact name of the file. The following example shows the creation of
the blackbox trace, and the trace info returned by the fn_trace_getinfo function:
DECLARE @rc int, @TraceID int
EXEC @rc = sp_trace_create @TraceID output, 8
SELECT TraceID = @traceID, error = @rc

EXEC sp_trace_setstatus @traceID, 1

SELECT * from ::fn_trace_getinfo(@TraceID)

RESULTS:
TraceID error
----------- -----------
1 0

(1 row(s) affected)

traceid property value

----------- ----------- --------------------------------------
1 1 8
1 2 C:\Program Files\Microsoft SQL Server
\MSSQL\data\blackbox
1 3 5
1 4 NULL
1 5 1

(5 row(s) affected)

If blackbox recording has been enabled, running sqldiag will copy the blackbox.trc file to sqldiag.trc in the same
directory as the sqldiag.txt file. If you specify a name other than sqldiag for the text file, the blackbox file will also be
copied to the \LOG directory with a different name. For example, if you specify a sqldiag output filename as
MyReport.txt, the blackbox trace file will be renamed to MyReport.trc. I mentioned earlier that even if SQL Server is
not running when sqldiag is executed, the blackbox file will still be available. This can be very useful if SQL Server
came down because of an error; the blackbox trace file will contain a record of what queries were executed
immediately before the failure. You might consider creating a stored procedure to enable blackbox tracing, and mark
that procedure for autostart. In this case, you'll know that you'll always have a record of the queries that were
executed, any time an unexpected failure occurs. On the companion CD, I have included a script to create such a
procedure and mark the procedure for autostart.

When you first enable blackbox tracing, you should see the blackbox.trc file in the \Data directory, but you might
notice that the size of file looks like it is staying at 0. In fact, you might not actually see a positive size reported for the
blackbox.trc file until one of the following occurs:
 The size of the trace exceeds 5 MB.

The blackbox trace is automatically configured for a maximum file size of 5 MB and also configured to
rollover and start a new file called blackbox_1.trc when the initial 5 MB is exceeded. If this second trace
exceeds 5 MB in size, the trace reverts to the initial blackbox.trc file. Until the first rollover occurs, the
operating system file might show a size of 0, even though trace information is being collected.
This document is created with the unregistered version of CHM2PDF Pilot

 You execute sqldiag.

When you run sqldiag, and the trace has more than 128 KB of data, the blackbox trace information will be
written to the blackbox.trc file on disk in the \Data directory and copied to the \LOG directory as sqldiag.trc.
Note that while the trace is running, information is written to the blackbox.trc file in 128 KB chunks, so that
the trace file might not contain the most recent events. If you need the blackbox.trc file or the sqldiag.trc file to
have all the events up to a specific time, you must use one of the following two options.
o The blackbox trace is stopped and removed.

When the blackbox trace is closed, SQL Server writes the contents to the blackbox.trc file. If you
subsequently run sqldiag, the complete blackbox.trc file is copied to the \Data directory as sqldiag.trc. If
you know the trace ID number, you can use the sp_trace_setstatus procedure to both stop and remove
the trace, as shown in these examples:
EXEC sp_trace_setstatus <TraceID>, 0 -- Stop the trace

EXEC sp_trace_setstatus <TraceID>, 2 -- Remove the trace

o You stop SQL Server.

Shutting down SQL Server effectively stops and closes the trace. This means that if SQL Server stops
because of an error condition, and you have previously enabled blackbox tracing, you will have a record
of the events just before SQL Server stopped.

To enable blackbox tracing to start automatically when SQL Server starts, you must create the trace in stored
procedure. Other traces can also be defined inside stored procedures, even if you don't mark them for autostart. You
might consider creating a number of procedures that capture events relevant to different performance conditions. For
example, one trace might capture events related to stored procedure recompilation; another trace might capture
information relating to automatic growth of a database and another might capture deadlock information. You could
then set up an alert to watch related Performance Counters using SQL Agent, and when a certain situation is
encountered or a threshold is crossed, you could start a trace to gather more data. You'll probably want to limit the
size of the trace file and not allow it to roll over and start a new file.

Tracing and Auditing

C2 auditing is necessary if you are running a C2 certified system. A C2 certified system meets a strict government
standard that defines the database security level. To have a C2 certified SQL Server, you must configure SQL Server
in the evaluated C2 configuration. You must enable c2 audit mode as a server-wide configuration option, and this will
automatically start tracing a large number of events specifically designed for auditing purposes. If you look at the list of
possible events, you'll see several dozen of them are in the Security Audit category. For more information about C2
certification, see the C2 Administrator's and User's Security Guide, which you can find at
http://www.microsoft.com/downloads/release.asp?ReleaseID=24481.
Performance Considerations
SQL Server tracing uses a "pay as you go" model. No overhead is incurred for events that are not captured. Most
events need very few resources. SQL Profiler becomes expensive only if you trace all of the more than 100 event
classes and capture all data from those events. Early testing shows an absolute maximum of 5 to 10 percent overhead
if you capture everything. Most of the performance hit is due to a longer code path; the actual resources needed to
capture the event data are not particularly CPU intensive. In addition, to minimize the performance hit, you should
define all your traces as server-side traces, and avoid the overhead of producing row sets to send to the SQL Profiler
client.

System Monitor
You can use SQL Profiler to monitor the behavior of your server on a query-by-query or event-by-event basis. But
This document is created with the unregistered version of CHM2PDF Pilot

if you want to monitor your server's performance as an entire system, the best tool is System Monitor. This tool is
extensible, which allows SQL Server to export its performance statistics so that you can monitor your entire system.
That's crucial because such important statistics as CPU use must be monitored for the entire system, not just for SQL
Server. In fact, many of the most important counters to watch while performance tuning SQL Server don't even
belong to any of the SQL Server objects.

System Monitor comes with your Windows 2000 operating system. In Windows NT, the equivalent tool was called
Performance Monitor. The discussion here will be relevant no matter which version you are using. Traditionally, in
mainframe and minicomputer systems, you had to buy a separate system monitor—at considerable expense—or use a
hodgepodge of utilities, some to monitor the operating system and others to monitor the database. Then, if you
worked on some other system, it didn't have the same set of tools you were accustomed to using.

In the Programs/Administrative Tools folder, click on the Performance icon to start System Monitor.

System Monitor provides a huge set of counters. Probably no one understands all of them, so don't be intimidated.
Peruse all the objects and counters and note which ones have separate instances. Don't confuse System Monitor's use
of the word "instance" with a SQL Server 2000 instance. System Monitor uses instance to refer to a particular
occurrence of a counter. For example, the SQLServer:Databases object has counters for things like number of active
transactions, log file size, and transactions per second. But since there are lots of different databases in a SQL Server
instance (such as master, Northwind, and pubs), you also need to choose a particular database from the instance list.
Other counters, such as SQLServer:Memory Manager apply to the entire SQL Server installation, and there are no
separate instances to deal with. If you want to monitor separate SQL Server instances, there are separate counters to
use. For example, the SQLServer:Memory Manager object, which I just mentioned, pertains only to the default
instance of SQL Server on the machine. If I also have a named instance called ENT2, there will be an entire set of
objects and counters for that instance, including an object called MSSQL$ENT2:Memory Manager. For more
information about performance counters, click the Explain button in the Select Counters dialog box, or see the SQL
Server and operating system documentation.
SEE ALSO
For general information about using System Monitor, look at the Windows 2000 Server Operations Guide in the
Windows 2000 Server Resource Kit, particularly chapters 5 through 10. If you're running Performance Monitor on
NT 4, you can look at the Windows NT Workstation Resource Kit Version 4.0 (Chapters 9 through 16). (Prior to
Windows NT 4, the Performance Monitoring section of the resource kit was contained in a separate volume called
Optimizing Windows NT by Russ Blake, and this has lots of information that is still very useful.)

Performance Monitor Counters

In this section, we'll look at several important counters. I'll briefly explain how they can be useful and what actions
you should consider based on the information they provide. Often, of course, the appropriate action is generic: for
example, if CPU usage is high, you should try to reduce it. The methods you can use to make such adjustments are
varied and vast—from redesigning your application to reworking some queries, adding indexes, or getting faster or
additional processors.

Object: Processor

Counter: % Processor Time

This counter monitors systemwide CPU usage. If you use multiple processors, you can set up an instance for each
processor. Each processor's CPU usage count should be similar. If not, you should examine other processes on the
system that have only one thread and are executing on a given CPU. Ideally, your system shouldn't consistently run
with CPU usage of 80 percent or more, although short spikes of up to 100 percent are normal, even for systems with
plenty of spare CPU capacity. If your system runs consistently above 80 percent or will grow to that level soon, or if
it frequently spikes above 90 percent and stays there for 10 seconds or longer, you should try to reduce CPU usage.

First, consider making your application more efficient. High CPU usage counts can result from just one or two
This document is created with the unregistered version of CHM2PDF Pilot

problematic queries. The queries might get high cache-hit ratios but still require a large amount of logical I/O. Try to
rework those queries or add indexes. If the CPU usage count continues to be high, you might consider getting a faster
processor or adding processors to your system. If your system is running consistently with 100 percent CPU usage,
look at specific processes to see which are consuming the CPUs. It's likely that the offending process is doing some
polling or is stuck in a tight loop; if so, the application needs some work, such as adding a sleep.

You should also watch for excessively low CPU usage, which indicates that your system is stalled somewhere. If
locking contention is occurring, your system might be running at close to 0 percent CPU usage when no productive
work is happening! Very low CPU usage can be a bigger problem than very high CPU usage. If you have poor
overall throughput and low CPU usage, your application has a bottleneck somewhere and you must find and clear it.

Note that Task Manager (Ctrl-Shift-Escape) in Windows NT 4 and Windows 2000 also provides a way to monitor
CPU usage. Task Manager is even easier to use than System Monitor.

Object: PhysicalDisk

Counter: Disk Transfers/sec

This counter shows physical I/O rates for all activity on the machine. You can set up an instance for each physical
disk in the system or watch it for the total of all disks. SQL Server does most I/O in 8 KB chunks, although
read-ahead I/O is essentially done with an I/O size of 64 KB. Watch this counter to be sure that you're not maxing
out the I/O capacity of your system or of a particular disk. The I/O capacity of disk drives and controllers varies
considerably depending on the hardware. But today's typical SCSI hard drive can do 80 to 90 random 8 KB reads
per second, assuming that the controller can drive it that hard. If you see I/O rates approaching these rates per drive,
you should verify that your specific hardware can sustain more. If not, add more disks and controllers, add memory,
or rework the database to try to get a higher cache-hit ratio and require less physical I/O (via better design, better
indexes, possible denormalization, and so on).

To see any of the counters from the PhysicalDisk object, you must reboot your computer with the Diskperf service
started. You do this from a command prompt, with the command DISKPERF. Using the -Y option sets the system to
start all the disk performance counters when the system is restarted. After you're done monitoring, disable Diskperf.

Object: PhysicalDisk

Counter: Current Disk Queue Length

This counter indicates the number of reads that are currently outstanding for a disk. Occasional spikes are OK,
especially when an operation that generates a lot of asynchronous I/O, such as a checkpoint, is activated. But for the
most part, the disks should not have a lot of queued I/O. Those operations, of course, must ultimately complete, so if
more than one operation is queued consistently, the disk is probably overworked. You should either decrease
physical I/O or add more I/O capacity.

Object: Memory

Counter: Pages/sec and Page Faults/sec

These counters watch the amount of paging on the system. As the system settles into a steady state, you want these
values to be 0—that is, no paging going on in the system. In fact, if you allow SQL Server to automatically adjust its
memory usage, it will reduce its memory resources when paging occurs. You should find that any paging that does
occur is not due to SQL Server. If your system does experience regular paging, perhaps due to other applications
running on the machine, you should consider adding more physical memory.

Object: Process

Counter: % Processor Time

This document is created with the unregistered version of CHM2PDF Pilot

Typically, you run this counter for the SQL Server process instance, but you might want to run it for other processes.
It confirms that SQL Server (or some other process) is using a reasonable amount of CPU time. (It doesn't make
much sense to spend a lot of time reducing SQL Server's CPU usage if some other process on the machine is using
the larger percentage of the CPU to drive the total CPU usage near capacity.)

Object: Process

Counter: Virtual Bytes

You use this counter to see the total virtual memory being used by SQL Server, especially when a large number of
threads and memory are being consumed. If this number gets too high, you might see Out Of Virtual Memory errors.

Object: Process

Counter: Private Bytes

This counter shows the current number of bytes allocated to a process that cannot be shared with other processes. It
is probably the best System Monitor counter for viewing the approximate amount of memory committed by any
threads within the sqlservr.exe process space. Additional SQL Server 2000 instances will be labeled sqlservr#1,
sqlservr#2, and so on.

Object: Process

Counter: Working Set

This counter shows the amount of memory recently used by a process. For a SQL Server process instance, this
counter can actually be a valuable indicator of how much memory has been allocated within the SQL Server process
space, especially for a dedicated server (because working-set trimming will likely not happen much). The value
recorded by the working set should be very close to the value reported in Task Manager, on the Processes tab, as
the Mem Usage value for the instance of sqlservr.exe. Working Set is the current memory that SQL Server (and any
components loaded in it) is currently accessing. It might not reflect the total amount of memory that SQL Server (and
any component loaded in its process space) has allocated. Here, we use the term "allocated" to mean memory that
has been committed to SQL Server. As long as no trimming has occurred, Working Set is the best counter for seeing
how much memory has been allocated within the SQL Server process space.

Process: Private Bytes (described above) does not show all the memory committed, but the value can be more stable
than Working Set if trimming occurs. You should consider monitoring both Private Bytes and Working Set. If you see
Working Set dip below Private Bytes, you should look at Private Bytes. If Working Set dips below Private Bytes, the
operating system must be trimming SQL Server's Working Set. This means other processes are competing for
memory that SQL Server might need to get back, and you should evaluate what other processes are competing with
SQL Server for memory resources.

Object: SQLServer:Buffer Manager

Counter: Buffer Cache Hit Ratio

There is no right value for the buffer cache-hit ratio because it is application specific. If your system has settled into a
steady state, ideally you want to achieve rates of 90 percent or higher, but this is not always possible if the I/O is
random. Keep adding more physical memory as long as this value continues to rise or until you run out of money.

Object: SQLServer:Memory Manager

Counter: Total Server Memory (KB)

This document is created with the unregistered version of CHM2PDF Pilot

This counter can be useful, but it doesn't reflect all memory allocated within a SQL Server process space. It reflects
only memory allocated in the SQL Server buffer pool. Note that the buffer pool is not just for data pages; it is also for
other memory allocations within the server, including plans for stored procedures and for ad hoc queries. Certain
components can be loaded into the SQL Server process space and allocate memory that is not under SQL Server's
direct control. These include extended stored procedures, OLE Automation objects, and OLE DB provider DLLs.
The memory space needed for these types of objects is included in SQL Server's Working Set but not in the Total
Server Memory counter.

Object: SQLServer:Cache Manager

This object contains counters for monitoring how the cache is being used for various types of objects, including ad
hoc query plans, procedure plans, trigger plans, and prepared SQL plans. A separate Cache Hit Ratio counter is
available for each type of plan, as is a counter showing the number of such objects and the number of pages used by
the objects.

Object: SQLServer:Buffer Manager

Counter: Page Reads/sec

Watch this counter in combination with the counters from the PhysicalDisk object. If you see rates approaching the
capacity of your hardware's I/O rates (use 80 to 90 I/Os per disk per second as a guide), you should reduce I/O
rates by making your application more efficient (via better design, better indexes, denormalization, and so on). Or you
can increase the hardware's I/O capacity. This statistic measures only read operations, not writes, and it does so only
for SQL Server, so you don't see the whole picture.

Object: SQLServer:Buffer Manager

Counter: Page Writes/sec

This value keeps track of all physical writes done by SQL Server, for any reason. It includes checkpoint writes,
lazywriter writes, and large block writes done during index creation or bulk copy operations. Separate counters are
available for checkpoint writes and lazywriter writes; you're probably better off using these more specific counters.

Object: SQLServer:Databases

Counter: Log Flushes/sec

This value should be well below the capacity of the disk on which the transaction log resides. It is best to place the
transaction log on a separate physical disk drive (or on a mirrored drive) so that the disk drive is always in place for
the next write, since transaction log writes are sequential. There is a separate counter for each database, so make sure
you're monitoring the right database instance.

Object: SQLServer:Databases

Counter: Transactions/sec

This counter measures actual transactions—either user-defined transactions surrounded by BEGIN TRAN and
COMMIT TRAN or individual data modification statements if no BEGIN TRAN has been issued. For example, if
you have a batch that contains two individual INSERT statements, this counter records two transactions. The counter
has a separate instance for each database and there is no way to keep track of total transactions for all of SQL
Server. Use it only as a general indication of your system's throughput. There is obviously no "correct" value—just the
higher the better.

User-Defined Counters
This document is created with the unregistered version of CHM2PDF Pilot

SQL Server 2000 offers 10 performance monitor counters for keeping track of any data or information that is useful
to you. These counters are under the SQL Server:User Settable object. The only counter is called Query, and there
are 10 instances to choose from. Ten stored procedures are available for specifying a value that System Monitor will
chart for a particular instance. For example, if you want to use User Counter 1 to keep track of how many rows are
in the invoices table, you can create triggers on the invoices table so that every time a row is inserted or deleted, the
stored procedure sp_user_counter1 is executed. You can include the following code in the triggers:
DECLARE @numrows int
SELECT @numrows = COUNT(*) FROM invoices
EXEC sp_user_counter1 @numrows

Once it is assigned a value, the user counter maintains that value until a new value is assigned. Note that the value
passed to the stored procedure must be a constant, a variable, or a parameterless system function starting with @@,
and its datatype must be Integer.

Other Performance Monitor Counters

The counters mentioned in this chapter are just the tip of the iceberg. SQL Server has dozens of others, for
monitoring almost every aspect of SQL Server behavior discussed in this book. The best way to learn about them is
to experiment with System Monitor. Look through the list of available objects and counters, select a few for viewing,
and see what results you get. You'll probably find a few that you'll want to use all the time, and others that you'll be
interested in only occasionally. But do revisit the complete list because there are too many to remember. A month
from now, you'll rediscover a counter that you completely overlooked the first time. You should also revisit the
available objects and counters after every service pack upgrade because new ones might be added as the SQL
Server development team gets feedback.

Tables 17-1 through 17-5 show some of the counters you might want to experiment with; the descriptions are
adapted from the online documentation.

Table 17-1. Counters for the SQLServer:Access Methods object.

Counter Description
Number of extents allocated per second to database
Extents Allocated/sec
objects used for storing index or data records.
Number of records per second fetched through
Forwarded Records/sec
forwarded record pointers.
Number of unrestricted full scans per second. These can
Full Scans/sec
be either base-table or full-index scans.
Number of index searches per second. These are used to
Index Searches/sec start range scans and single-index record fetches and to
reposition an index.
Number of page splits per second that occur as the result
Page Splits/sec
of overflowing index pages.
Number of pages allocated per second to database
Pages Allocated/sec
objects used for storing index or data records.
Number of probe scans per second. These are used to
Probe Scans/sec
find rows in an index or base table directly.
Number of qualified range scans through indexes per
Range Scans/sec
second.
Number of ghosted records per second skipped during
Skipped Ghosted Records/sec
scans.
Table Lock Escalations/sec Number of times locks on a table were escalated.
This document is created with the unregistered version of CHM2PDF Pilot

Worktables Created/sec Number of worktables created per second.

Table 17-2. Counters for the SQLServer:Databases object.

Counter* Description
Active Transactions Number of active transactions for the database.
Bulk Copy Rows/sec Number of rows bulk copied per second.
Bulk Copy Throughput/sec Amount (in kilobytes) of data bulk copied per second.
Cumulative size (in kilobytes) of all the data files in the
database, including any automatic growth. This counter is
Data File(s) Size (KB)
useful for determining the correct size of tempdb, for
example.
Percentage of log cache reads satisfied from the log
Log Cache Hit Ratio
cache.
Cumulative size (in kilobytes) of all the transaction log files
Log File(s) Size (KB)
in the database.
Total number of times the transaction log for the database
Log Growths
has been expanded.
Total number of times the transaction log for the database
Log Shrinks
has been shrunk.
Total number of times the transaction log for the database
Log Truncations
has been truncated.
Percent Log Used Percentage of space in the log that is in use.
Amount of data being moved per second by autoshrink
Shrink Data Movement Bytes/sec operations or by DBCC SHRINKDATABASE or
DBCC SHRINKFILE statements.
Number of transactions started for the database per
Transactions/sec
second.

* Separate instances of these counters exist for each database.

Table 17-3. Counters for the SQLServer:Locks object.

Counter Description
Average amount of wait time (in milliseconds) for each
Average Wait Time (ms)
lock request that resulted in a wait.
Number of new locks and lock conversions per second
Lock Requests/sec
requested from the lock manager.
Number of lock requests per second that timed out,
Lock Timeouts/sec
including internal requests for NOWAIT locks.
Total wait time (in milliseconds) for locks in the last
Lock Wait Time (ms)
second.
Number of lock requests per second that could not be
Lock Waits/sec
satisfied immediately and required the caller to wait.
Number of lock requests per second that resulted in a
Number of Deadlocks/sec
deadlock.

Table 17-4. Counters for the SQLServer:Memory Manager object.

Counter Description
Total amount of dynamic memory the server is using for
Connection Memory (KB)
maintaining connections.
This document is created with the unregistered version of CHM2PDF Pilot

Total amount of dynamic memory the server is using for

Lock Memory (KB)
locks.
Maximum amount of memory available for executing
Maximum Workspace Memory (KB) processes such as hash, sort, bulk copy, and index
creation operations.
Total number of processes per second that have
Memory Grants Outstanding
successfully acquired a workspace memory grant.
Total number of processes per second waiting for a
Memory Grants Pending
workspace memory grant.
Total amount of dynamic memory the server is using for
Optimizer Memory (KB)
query optimization.
Total amount of dynamic memory the server is using for
SQL Cache Memory (KB)
the dynamic SQL cache.
Total amount of dynamic memory the server is willing to
Target Server Memory (KB)
consume.
Total amount (in kilobytes) of dynamic memory that the
Total Server Memory (KB)
server is currently using.

Table 17-5. Counters for the SQLServer:SQL Statistics object.

Counter Description
Number of autoparameterization attempts per second.
Total should be the sum of the failed, safe, and unsafe
autoparameterizations. Autoparameterization occurs when
Auto-Param Attempts/sec
SQL Server attempts to reuse a cached plan for a
previously executed query that is similar to, but not
exactly the same as, the current query.
Number of Transact-SQL command batches received
Batch Requests/sec
per second.
Number of failed autoparameterization attempts per
Failed Auto-Params/sec
second. This should be small.
Number of safe autoparameterization attempts per
Safe Auto-Params/sec
second.
Number of SQL compilations per second. Indicates the
number of times the compile code path is entered.
SQL Compilations/sec Includes compiles due to recompiles. Once SQL Server
user activity is stable, this value should reach a steady
state.
Number of unsafe autoparameterization attempts per
second. The table has characteristics that prevent the
Unsafe Auto-Params/sec
cached plan from being shared. These are designated as
unsafe. The fewer of these that occur the better.

Other Performance Monitoring Considerations

Any time you monitor performance, you also slightly alter performance simply because of the overhead cost of
monitoring. It can be helpful to run System Monitor on a separate machine from SQL Server to reduce that overhead.
The SQL Server-specific counters are obtained by querying SQL Server, so they use connections to SQL Server.
(System Monitor always uses Windows Authentication to connect to the SQL Server.) It is best to monitor only the
counters you are interested in.
This document is created with the unregistered version of CHM2PDF Pilot

34

Summary
Configuration alone can rarely turn your slow, plodding application into a screamer. In fact, you're more likely to
misconfigure your system by taking manual control of the options that SQL Server sets automatically. Knowing what
each configuration option does can keep you from misconfiguring your system.

In this chapter, we looked at the configuration options and discussed some guidelines for their use. I also discussed
how to monitor the effects on your system's performance using SQL Profiler, server traces and System Monitor. SQL
Server tracing lets you determine which queries, procedures, or users have the biggest performance impact. System
Monitor gives you a scientific method of holding constant all but one variable within the system as a whole so that you
can measure the effect as you vary it.
This document is created with the unregistered version of CHM2PDF Pilot

34

Bibliography and Suggested

Reading
Despite this book's considerable length, it doesn't touch on every important area relating to Microsoft SQL Server,
and it assumes that readers have some knowledge of relational database management systems (RDBMSs). This
section includes other recommended resources for learning about the SQL language, the fundamentals of using
Microsoft SQL Server, and related technologies. However, the most important recommendation I can make is to
read, cover to cover, the SQL Server documentation.

One of the CDs included with this book contains an evaluation copy of the complete SQL Server 2000 product. To
access to the product documentation, you can run the setup program and choose to install only the Books Online
from the Client Utilities. The other CD contains some valuable whitepapers. You should start there. This second CD
also contains some tools for use with SQL Server as well as an online Transact-SQL language reference compiled by
a member of the SQL Server development team. Although the reference is a Microsoft Word file, it contains many
cross-linked references and is intended for online use.

You can augment your reading with the following books and other resources. I've listed them in order from
introductory materials to the more detailed or specialized materials.

SQL (Osborne's Complete Reference Series) by James R. Groff and Paul N. Weinberg (Osborne McGraw-Hill,
1999). An excellent primer for users who are new to SQL.

An Introduction to Database Systems, 7th Edition, by C. J. Date (Addison-Wesley, 1999). A new revision of a
classic book, written by a giant in the field, that covers general relational database concepts. A must-read for
everyone in the database industry.

Microsoft SQL Server 7.0 Database Implementation Training Kit and Microsoft SQL Server 7.0 System
Administration Training Kit (Microsoft Press, 1999). Self-paced training with plenty of information about
administration and security. These kits are fairly introductory and are suitable for those newer to SQL Server. The
1999 edition is based on SQL Server 7. The SQL Server 2000 version should be available soon. Check the
Microsoft Press Web site (http://mspress.microsoft.com) periodically for details.

Database Design for Mere Mortals by Michael J. Hernandez (Addison-Wesley, 1997). A very readable
approach to the daunting task of designing a relational database to solve real-world problems. The book is written in
a database-independent format, so the details can be applied to SQL Server as well as to other RDMSs you might
use.

Handbook of Relational Database Design by Candace C. Fleming and Barbara Vonhalle (Addison-Wesley,
1988). A fine book that discusses general logical database design and data modeling approaches and techniques.

SAMS Teach Yourself Microsoft SQL Server 2000 in 21 Days by Richard Waymire and Rick Sawtell (SAMS
Publishing, 2000). A good first book on SQL Server 2000 in tutorial format that gets readers up to speed quickly.
One of the coauthors is a program manager on the SQL Server development team. This title includes self-test
questions for each of the 21 "days."

Database: Principles, Programming, Performance by Patrick O'Neil (Morgan Kaufmann Publishers, 1994).
This thorough textbook provides excellent introductory materials, so I've placed it high on the list. It also carefully
details a broad spectrum of database topics, including buffer management and transaction semantics.
This document is created with the unregistered version of CHM2PDF Pilot

Understanding the New SQL: A Complete Guide by Jim Melton and Alan R. Simon (Morgan Kaufmann
Publishers, 2000). Melton is an active participant in the SQL standards work and was the editor of the ANSI
SQL-92 standard. Although you can get the ANSI SQL specification directly from ANSI, this book translates the
standard into English.

SQL-99 Complete, Really by Peter Gulutzan and Trudy Pelzer (R&D Books, 1999). An excellent reference that I
consult frequently for issues regarding ANSI SQL-99 semantics and conformance.

A Guide to the SQL Standard by C. J. Date with Hugh Darwen (Addison-Wesley, 1997). Similar in purpose and
focus to the Melton and Simon book, this book offers more compact coverage, provides additional insight into why
things work the way they do, and includes more discussion of semantics. It is an especially good reference on the use
of NULL.

Optimizing Transact-SQL: Advanced Programming Techniques by David Rozenshtein, Anatoly Abramovich,

and Eugene Birger (Coriolis Group, 1997). A book that offers lots of clever queries and solutions written with
Transact-SQL. (Note, however, that the solutions here were authored before SQL Server had the CASE construct,
and CASE can sometimes provide easier, more straightforward solutions.)

Joe Celko's SQL for Smarties: Advanced SQL Programming by Joe Celko (Morgan Kaufmann Publishers,
1999). An excellent book for insight into subtle but powerful ways to write nonintuitive queries. In places, this one is
truly the SQL book for the Mensa crowd—it's loaded with mind-bending puzzles about writing SQL queries to
perform nonobvious tasks. It contains many examples—probably more than any other book on this list—and you're
likely to find solutions to problems similar in scope to ones you might face.

Hitchhiker's Guide to Visual Basic and SQL Server by William Vaughn (Microsoft Press, 1998). A good book
that discusses the various programming interfaces available through Microsoft Visual Basic, especially when you're
accessing SQL Server.

Inside ODBC by Kyle Geiger (Microsoft Press, 1995). The finest book available on ODBC, period. Written by the
"Father of ODBC." A must for C programmers who work with ODBC and SQL Server. It includes a fine discussion
of cursors as well. Although this book is out of print, you might know someone who has a copy or you can check at
used bookstores.

SQL Server: Common Problems, Tested Solutions by Neil Pike (APress, 2000). An expanded FAQ list,
originally compiled by SQL Server MVP Pike, with contributions by most of the other SQL Server MVPs. Lots of
details on frequently confusing issues that arise in working with all versions of SQL Server, from 6 to 2000. The
solutions are either undocumented or poorly explained in the product documentation. Some answers are very short,
others are pretty extensive, depending on the question. The book provides many possible answers and ways of doing
things.

Microsoft Developer Network (MSDN) Library. Various articles and resources (available online at
http://msdn.microsoft.com/library/default.asp) from Microsoft. Like Microsoft KnowledgeBase, the MSDN library
contains information about all the Microsoft development products. A must for serious developers.

Professional SQL Server 7.0 Programming by Rob Vieira (WROX Press, 1999). A book that has it all, from
basic database design and stored procedures to advanced queries and linked servers to advanced topics such as
performance tuning and security. It also has a solid description of OLAP Services and Data Transformation Services.
A SQL Server 2000 version of this book is still in the works.

The Guru's Guide to Transact-SQL by Ken Henderson (Addison Wesley, 1999). A book that offers the most
complete set of code examples I have seen—over 600 of them! These examples not only illustrate important concepts
and best practices, but they can be incorporated into real-world applications. Henderson has a very good discussion
of SOUNDEX and a proposal for an alternative implementation. The book covers basic material as well as advanced
topics such as OLE automation and full-text searching. It was because of this book that I finally started to understand
This document is created with the unregistered version of CHM2PDF Pilot

and get excited about the power of full-text searching and the CONTAINSTABLE function.

Advanced Transact-SQL for SQL Server 2000 by Itzik Ben-Gan and Tom Moreau (APress, 2000). A book by
two writers who really know Transact-SQL—in fact, they probably dream in T-SQL. Anything written by Ben-Gan
is guaranteed to be a gold mine. Ben-Gan and Moreau provide lots of details on new SQL Server 2000 features such
as user-defined functions (UDFs), indexed views, cascading actions, federated views, and hierarchical structures.

Microsoft SQL Server 7.0 Performance Tuning Technical Reference by Steve Adrien Deluca (Editor),
Marcilina S. Garcia, Jamie Reding, and Edward Whalen (Microsoft Press, 2000). Provides information that database
administrators need in order to configure a SQL Server 7 RDBMS for optimum performance and scalability. Unlike
many tuning books on the market, it teaches not just about SQL Server but also about the underlying operating
system and hardware. A SQL Server 2000 version is in the works.

Performance Tuning and Optimization of Microsoft SQL Server 7.0 (Microsoft Authorized Training Course
#2013). Recommended for people who are interested in becoming the SQL Server performance expert on their
team. This course is available from authorized training centers. A separate SQL Server 2000 version of this course
won't be offered because much of the relevant information has been included in this course and in the other SQL
Server 2000 curricula. Check the Microsoft training and certification Web site (http://www.microsoft.com/train_cert)
periodically for details.

Transaction Processing: Concepts and Techniques by Jim Gray and Andreas Reuter (Morgan Kaufmann
Publishers, 1992). Without a doubt, the best book with in-depth explanations of such issues as locking, logging and
recovery, and transactional semantics. This book is unusual (and terrific!) in that it is packed with detailed information
but is still readable and understandable—unlike many of the overly academic papers that seem more suitable for
graduate school classes than for people trying to build actual solutions.

Principles of Database Query Processing for Advanced Applications by Clement T. Yu and Weiyi Meng
(Morgan Kaufmann, 1998). For those who want to know what happens under the hood. This is a textbook-style
guide to an important area of database theory—how queries work in a variety of databases, and the strategies and
algorithms used to optimize them. The book covers topics relevant to relational databases, object-oriented databases,
and distributed databases.

In Search of Clusters: The Ongoing Battle in Lowly Parallel Computing by Gregory F. Pfister (Prentice
Hall, 1998). A good source of information on scalability limits of symmetric multiprocessing (SMP) systems, such as
Amdahl's law, the Von Neumann bottleneck, and other considerations for SMP systems.

I'd also like to recommend the following resources:

Transaction Processing Council (TPC) Web site and full disclosure reports (FDRs). Invaluable resources
available at http://www.tpc.org. The Web site allows you to stay abreast of current performance levels of competing
hardware and software solutions. It also has plenty of information regarding benchmarks. You also might want to
order one of the FDRs for a benchmark that uses SQL Server to determine the exact hardware and software
configuration used. If you plan to run a very large system, such an FDR can be a good template to use.
This document is created with the unregistered version of CHM2PDF Pilot

34

Footnotes
Chapter 1
1. Gray, Helland, O'Neil, and Shasha, "The Dangers of Replication and a Solution," SIGMOD (1996). SIGMOD
(Special Interest Group on Management of Data) is a yearly database-oriented conference for developers. For more
information, see http://www.acm.org/sigmod/.
This document is created with the unregistered version of CHM2PDF Pilot

34

Kalen Delaney
Kalen Delaney has worked with SQL Server extensively since joining the technical support group at Sybase in 1987,
shortly after the company's first public product release. Kalen soon began teaching Sybase's Performance and Tuning
course. She received the company's Instructor of the Year Award in 1990. Kalen left Sybase in 1992 to become an
independent Microsoft SQL Server trainer and consultant. Kalen has been the primary Subject Matter Expert (SME)
for two Microsoft SQL Server courses and has helped Microsoft develop training materials for bringing their product
support specialists up to speed on the new capabilities and internal behavior of new versions of SQL Server. She
teaches these special classes to SQL Server support engineers throughout the United States and around the world.

Kalen has worked more than 22 years in computer and database systems since earning her M.S. degree in computer
science at the University of California, Berkeley. She has published numerous articles in SQL Server Professional
and Windows 2000 Magazine and currently is a contributing editor and regular columnist for SQL Server Magazine.
She lives with her husband and four children in the Pacific Northwest, just a ferryboat ride from Microsoft's Redmond
campus.
This document is created with the unregistered version of CHM2PDF Pilot



34

About This Electronic Book

Every effort has been made to ensure that the text of this eBook matches the text of the paper version as closely as
possible. For usability reasons, formatting changes may have been made to the eBook version.

“Expanding― Graphics
Many of the graphics shown in this book are quite large. To improve the readability of the book, reduced versions of
these graphics are shown in the text. To see a full-size version, click on the reduced graphic.

External Links
This eBook may contain links to Web sites. All hyperlinks within the text were valid at the time this eBook was
published. Due to the nature of the World Wide Web, we cannot guarantee that all links to Web sites are still valid
after the release date of the eBook.

Tell Us What You Think

We need to hear from you regarding your experience with our eBooks. Tell us what you like, don't like; which
features you use, and which features you would like to see in future versions of our eBooks. Send your comments to
[email protected]. Please note that technical support is not offered through this alias.
This document is created with the unregistered version of CHM2PDF Pilot

About Microsoft Press

Microsoft Press is a division of Microsoft Corporation and the leading source of comprehensive self-paced learning,
training, evaluation, and support resources to help everyone from developers to IT professionals to end users get the
most from Microsoft® technology. Choose from more than 250 current titles in print, multimedia, and network-ready
formats—learning solutions made by Microsoft, with the most timely and accurate information available. For more
information, visit http://mspress.microsoft.com/.