eXtremeDB Dot Net User Guide
eXtremeDB Dot Net User Guide
(c) 2001-2011 McObject LLC 22525 SE 64th Place, Suite 302 Issaquah, WA 98027 USA Phone: 425-888-8505 Fax: 425-888-8508 E-mail: [email protected] www.mcobject.com
Table of Contents
Chapter 1: Introduction ................................................................................................................................ 3 Overview ....................................................................................................................................................... 3 Syntax Conventions................................................................................................................................... 5 Chapter 2: The Database............................................................................................................................... 6 Defining a Database .................................................................................................................................. 6 Opening/Creating a Database ................................................................................................................... 9 Mixed Language (Java/C/C++) Access ..................................................................................................... 10 Closing/Destroying a Database ............................................................................................................... 11 Chapter 3: Create, Update, Delete ............................................................................................................. 12 Transactions ............................................................................................................................................ 12 Inserting a Record ................................................................................................................................... 12 Updating a Record .................................................................................................................................. 13 Deleting a Record .................................................................................................................................... 14 Chapter 4: Database Search ........................................................................................................................ 15 Search by Index Type .............................................................................................................................. 15 Initiating an exact-match Search ............................................................................................................ 15 Simple Iteration....................................................................................................................................... 17 Cursor.Search Options ............................................................................................................................ 17 Compound Index Search ......................................................................................................................... 18 Performing a JOIN by AUTOID ................................................................................................................ 19 Patricia Trie Index ................................................................................................................................... 19 RTree Index ............................................................................................................................................. 19
Chapter 1: Introduction
This User Guide occasionally references the eXtremeDB User Guide (Version 4.0) and the eXtremeDB Reference Manual. You can also download white papers from the website (http://www.mcobject.com) and review the sample source code provided with the example programs in the eXtremeDB package.
Overview
The concept behind the .Net API to eXtremeDB (hereafter eXtremeDB.Net) is to provide seamless integration of the eXtremeDB database system and the C# programming languages supported by the Microsoft .Net Common Language Runtime (CLR). The eXtremeDB.Net API is implemented in C# and C++ (object-oriented languages) and is manipulated with objects. The main responsibility of the database interface is to provide for convenient storage of persistent objects and to allow the developer to save and load them without extra effort. This objective is realized in eXtremeDB.Net by using the C# reflection mechanism. Reflection allows a program to get information about classes format at runtime, create instances of arbitrary classes, and set/get the value of their fields. eXtremeDB.Net only requires the programmer to specify a list of the persistent capable classes: i.e. classes whose instances can be stored in the database. Then, when the application is executed, eXtremeDB.Net extracts information about these classes using the reflection mechanism and builds an eXtremeDB database dictionary to be passed into the eXtremeDB runtime library when the database is to be created or opened. The database dictionary is subsequently used by the eXtremeDB runtime to know the size of objects, what fields participate in what indexes, and so on, so that the reflection step is used only once. To provide efficient access to objects we also need indexes to be able to locate them by key value, or to iterate through them in the desired order. And, there are some database specific attributes that cannot be represented in standard C# class declaration. For example, eXtremeDB supports BLOBs (Binary Large OBjects), fixed size binary and character arrays, different string encodings, and other capabilities. All this non-standard class information should be provided by the developer directly in the class declaration using the C# annotation mechanism. eXtremeDB.Net supports all C# primitive types (int, boolean, float,...), strings and arrays. It is possible to define an arbitrary number of indexes for each class the developer need just annotate the fields that are keys. eXtremeDB .Net supports B-Tree, hash, R-Tree (spatial index) and Patricia Trie index types. Each index can be unique or allow duplicate key values. The uniqueness constraint, as well as sort order (ascending or descending) can be specified as a parameter to the Indexable annotation. eXtremeDBs notion of an AUTOID (a system-supplied unique identifier for an object; other DBMS might call this a serial, sequence or similar) can also be indicated in an
eXtremeDB .Net User Guide
annotation. If a class has no other index (this would be unusual), an eXtremeDB list index can be specified by the List annotation. The structure of eXtremeDB.Net is very simple and follows the C API of eXtremeDB closely. The main class is the Database which is responsible for opening and closing the database, and establishing connections with the database. Every thread of every process must have its own connection to the database. eXtremeDB supports multiple ways to create, use and share databases: in-memory database that is private to a single application (possibly with multiple threads) in-memory database shared between several applications (possibly with multiple threads each) mixed in-memory and on-disk database high availability database with one master and several replica nodes
An in-memory database consists of one or more memory segments (located in private process memory or in shared memory). An on-disk database also has one or more files for storing persistent objects, and a transaction log file. eXtremeDB database files can be regular OS files, raw partitions, multi-files and RAIDs. See the eXtremeDB Users Guide for more details. Once a database is opened, you must create a connection handle to the database. Each application thread must open its own connection. The database connection is represented in eXtremeDB.Net using the Connection class. This class provides methods for starting, committing and aborting transactions, and inserting and deleting objects. And the last eXtremeDB.Net class is Cursor. A cursor allows you to search or navigate through the objects using an index. When you construct a cursor, you specify the class and optionally, the name of the key field (a field that is marked with the indexable C# annotation). If the field name is not provided, then a list or AUTOOID index must exist for this class. Using the cursor, it is possible to iterate through all instances (objects) of the class associated with the cursor. Note that eXtremeDB does not support inheritance, and likewise persistent classes do not support inheritance. So if persistent class A is a base class for persistent class B, then a cursor for A will not iterate through instances of class B. Iterating through objects is great, but we also need to be able to efficiently locate objects by key value. The Cursor class provides two methods for this purpose: find and search. The find method can only be used for unique indexes, and it returns the instance of the located object or null if no object is found with that key value. The search method is more generic and with it, you can specify a search operation (the possible search operations depend on the type of index and includes such operations as equals, less-than, overlaps, ...) and iterate through the results of the search. The rules of iteration also depend on type of the index (see the eXtremeDB Users Guide for a complete explanation of cursor operations).
eXtremeDB .Net User Guide
For B-Tree indexes, eXtremeDB.Net positions the cursor at the first record matching the specified search criteria. For example, in the case of an equality comparison, it will be the first object with the specified value of the key. Moving the cursor forward iterates through all objects with that value of the key. Please note that the cursor will not stop when it reaches the last object with the specified value for the key. Instead, the cursor will advance and traverse through objects whose key value is greater than that specified. So for an exact match search, the application must compare the key value during iteration and break the loop when the object iterated to no longer has the specified search value. For Patricia trie and R-Tree indexes, the cursor will only iterate through the objects that match the search criteria - there is no need to perform extra checks. The eXtremeDB.Net Cursor class implements the MoveNext() method to iterate over all search results in a forward direction. But since eXtremeDB also supports iteration in the backward direction, the Cursor class also has MoveFirst(), MoveLast(),and MovePrev() methods.
Syntax Conventions
eXtremeDB.Net follows standard C# syntax conventions.
Defining a Database
The first step in using eXtremeDB.Net is to define the classes to be managed in the eXtremeDB database. eXtremeDB.Net uses the C# class syntax, with annotations. At run-time, C# reflection is used to discover the classes and their attributes and build an eXtremeDB database dictionary. Once created, the dictionary is subsequently used by the eXtremeDB runtime. Therefore, C# reflection is a one-time event when the database is first created or opened. The annotations that are recognized by eXtremeDB are as follows: Blob Uses eXtremeDB BLOB type for storing a byte array Blob public byte[] photo; specifies the dimension for fixed size arrays (char, scalar) An array without a dimension will be treated by eXtremeDB as a vector Examples: Dimension(8) String prefix; public float[] coord; // creates an eXtremeDB vector field specifies the encoding of string fields. Could be any valid encoding supported by the C# runtime. UTF-8 is the default. Examples: ("UTF-16") ("ASCII") include the field in an index (unique=true) creates a b-tree index (the default), enforces uniqueness (type=Database.IndexType.Hashtable, unique=true) creates a hash index (unique=false) creates a b-tree index that allows duplicates (type=Database.IndexType.Patricia) (type=Database.IndexType.RTree) (descending=true) indexes are ascending by default (initSize=N) for a hash index, where N is the initial size of the hash table
eXtremeDB .Net User Guide
Dimension
Encoding
Indexable
Indexes
Optional
Persistent
Define a compound (multi-field) index Indexes({ Index(name="byName", keys={ Key("lastName"), Key("firstName") }, unique=true ), Index(name="byAddress", keys={ Key("address.country"), Key("address.city"), Key("address.street") }, unique=false ), Index(name="bySalary", keys={ Key(value="salary", descending=true) } ) }) class Employee { String firstName; String lastName; Address address; long salary; } optional structure field Optional Address address; marks class as stored in the database and provides some attributes of this class Persistent(autoid=true) Persistent(list=true) Persistent(large=true) Persistent(compact=true) Persistent(inmemory=true) Persistent(disk=true)
Note that Persistent indicates that objects of a class should be stored in eXtremeDB. Whether objects are stored in-memory or on-disk (where disk means a file system, regardless of the physical media) is a function of these properties set in the C# class definition and the Database.Parameter: - if Parameters.diskClassesByDefault is set to true and Persistent(inmemory) is not specified, then objects of the class are disk based. - if Parameters.diskClassesByDefault is set to false and Persistent(disk) is not specified, then objects of the class are transient (in-memory). - if Persistent(inmemory) is set to true, then objects of the class are transient regardless of the value of Parameters.diskClassesByDefault - if Persistent(disk) is set to true, then objects of the class are persistent regardless of the value of Parameters.diskClassesByDefault A similar set of rules apply to Persistent(large) and Persistent(compact) and the related Parameters.compactClassesByDefault. The classes defined to be managed by eXtremeDB are passed into the eXtremeDB runtime as an array of objects of the class Class (see next section). C# classes that are not passed into the eXtremeDB run-time but are referenced within database classes will be treated as eXtremeDB structure fields. See the eXtremeDB Users Guide for an explanation of structure fields (and see the example /tests/csharp/TestDotNet).
Opening/Creating a Database
A database is represented in eXtremeDB.Net as an instance of the Database class:
Database db = new Database( new ExtremedbWrapper() );
(ExtremedbWrapper is a wrapper class necessary for the eXtremeDB managed API implemented in managed C++.) When opening or creating a database for the first time, a set of parameters must be established and passed into the eXtremeDB run-time. These parameters are set by a Database.Parameters object. Refer to the Database.Parameters page in the on-line reference (./doc/index.html) for the complete set. The following discussion focuses on the essential parameters:
Database.Parameters params = new Database.Parameters(); params.memPageSize = PAGE_SIZE; params.classes = new Class[]{Employee.class, Company.class};
For a database that is all, or part, on-disk, some additional parameters are required: params.diskPageSize = DISK_PAGE_SIZE; params.diskClassesByDefault = true; A Database will have a number of possible configurations that are enumerated in:
enum eXtremdDBConfiguration { MCO_CFG_DEBUG_LIBRARY = 1, MCO_CFG_MVCC_TRANSACTION_MANAGER = 2, MCO_CFG_SHARED_MEMORY = 4, MCO_CFG_DISK_SUPPORT = 8, MCO_CFG_HA_SUPPORT = 16 };
For example, to create an on-disk or hybrid database and use the debug version of the eXtremeDB run-time, instantiate the Database object as follows:
db = new Database( Database.MCO_CFG_DEBUG_LIBRARY | Database.MCO_CFG_DISK_SUPPORT);
To create an in-memory database and use the MVCC transaction manager (the MURSIW transaction manager is the default):
db = new Database(MCO_CFG_MVCC_TRANSACTION_MANAGER);
The next step is to open the database. The db.Open() method is overloaded and the arguments depend on whether youre creating an all in-memory database or an ondisk/hybrid database. For an all in-memory database:
eXtremeDB .Net User Guide
An in-memory database uses a set of default devices. An on-disk or hybrid database requires that you establish the array of database devices to be passed into the db.open() method. See the eXtremeDB Users Guide for a complete explanation of database devices. Briefly, an on-disk/hybrid database requires four devices. The first device can be either Database.PrivateMemoryDevice or Database.SharedMemoryDevice and is used by eXtremeDB for its internal meta data and to store in-memory objects (if there are any). It is of type Database.Device.Kind.Data. The second device establishes the cache for the on-disk database. It also can be either Database.PrivateMemoryDevice or Database.SharedMemoryDevice and is of type Database.Device.Kind.DiskCache. The third device represents the file that will contain the on-disk database. It must be a Database.FileDevice and be of type Database.Device.Kind.Data. The fourth and final device represents the eXtremeDB transaction log that supports recovery in the event of an abnormal termination of the program. It must be a Database.FileDevice and must be of type Database.Device.Kind.TransactionLog. A database should be created/opened only once in an application. Each thread within the application is required to have its own connection to the database. A database connection is represented by the Connection class, and is established by its constructor:
Connection con = new Connection(db);
db.generateMcoFile("test.mco");
A database used by eXtremeDB.Net is accessible through all programming languages supported by the .Net Common Language Runtime (CLR). The dynamic link library extremedb4net.dll provides the C# implementation of the eXtremeDB.Net classes. This dll must be included in the references for all .Net projects that interface with an eXtremeDB database. The eXtremeDB runtime internals are implemented in managed C++ code that cannot be mixed in the same assembly. So, in addition to extremedb4net.dll, an eXtremeDB.Net project must also include in its references the eXtremeDB libraries necessary for the given application, such as: extremedb4net_mursiw_debug.dll (the MURSIW Transaction Manager for conventional memory applications) or extremedb4net_disk_mursiw_debug.dll (the MURSIW Transaction Manager for disk-based applications). (See the sample projects in directory samples/csharp).
Closing/Destroying a Database
To close a database after making the necessary changes, the first disconnect from the database. The database, now a file stored in the platforms hard drive, is closed:
con.Disconnect(); db.Close();
To destroy or delete a database created and/or run within the eXtremeDB.Net framework, follow standard file deleting procedures for your target platform. The following directions use the Windows platform as an example: 1. Use Windows Explorer to locate the database file. 2. Right-click the file. 3. Click on Delete. The file will be sent to the Recycle Bin. Empty the Recycle Bin to completely destroy the file.
Transactions
All database activity must take place within the context of an eXtremeDB transaction. The eXtremeDB transaction manager is the component of eXtremeDB that is responsible for controlling concurrent access to the database, so any attempt to use the database in any way outside the scope of a transaction is illegal. To begin a transaction, the Connection class method StartTransaction is called:
con.StartTransaction(Database.TransactionType.ReadWrite);
The Database.TransactionType.ReadWrite argument readies the database to be modified. Transactions can also be Database.TransactionType.ReadOnly and Database.TransactionType.Exclusive. The the Connection class method CommitTransaction completes the transaction:
con.CommitTransaction();
Committing the transaction will cause the database run-time to update any indexes that were affected by the insert/update/delete actions, and make the changes made within the transaction visible to other connected threads and/or processes. A transaction can be abandoned by invoking the Connection class RollbackTransaction() method:
con.RollbackTransaction();
It is also possible to utilize eXtremeDBs two-phase commit, transaction priority, and transaction isolation options through the Connection class. See the eXtremeDB User Guide for a full explanation of these options.
Inserting a Record
After a transaction is successfully started, you can instantiate objects and assign values to their fields. For example:
Company company = new Company(); company.name = "McObject LLC";
The company object is created and inserted into the database via the Connection classs Insert method:
long companyID = con.Insert(company);
The insert method either returns zero or, if the class was annotated with (autoid=true), the generated AUTOID of the newly created object. As mentioned in the introduction, the AUTOID is often the easiest way to establish a relationship between two objects. For example, to associate a new employee with the company just inserted:
Employee employee = new Employee(); employee.name = "Hyman Farkleburger"; employee.age = 33; employee.salary = 50000; employee.companyID = companyID; // create relationship
The following line illustrates how to assign a value to an eXtremeDB BLOB type of field.
employee.photo = new byte[PHOTO_SIZE]; con.Insert(employee);
Updating a Record
To update a record, a ReadWrite transaction must be opened to locate the record to be updated. Locating and updating records are done through the Cursor class. A full explanation of database searches follows in the next section. In this section, just enough information about cursors is covered to explain the basics of locating an object for the purpose of further illustrating update operations. The simplest way to locate an object is by an exact match lookup through a hash or unique b-tree index (a b-tree index that does not allow duplicate key values). To begin, instantiate a Cursor object:
con.StartTransaction(Database.TransactionType.ReadWrite); Cursor<Employee> emplCursor = new Cursor<Employee>(con, Employee.class, name);
The preceding code instructs the system to create a cursor object for class Employee and names the variable emplCursor. The Cursor class constructor takes the database Connection, the class type specifier (Employee.class), and the name of the indexed field for which the cursor will be associated. In this case, it is the Employee.name field.
eXtremeDB .Net User Guide
And now one or more of the fields of the object can be updated:
employee.salary = employee.salary * 1.1;
Lastly, the Cursor.Update() method is used to write the changes to the object referenced by the cursor back into the database:
emplCursor.Update();
For further explanation of cursor operations and index types, please refer to the eXtremeDB User Guide.
Deleting a Record
eXtremeDB.Net provides methods to delete individual objects and to delete all the objects of a class. As with the update operation in the previous section, to delete a record, first begin a ReadWrite transaction, then locate the record to be deleted:
con.StartTransaction(Database.TransactionType.ReadWrite); employees = new Cursor<Employee>(con, Employee.class); nEmployees = 0; foreach (Employee e in employees) { nEmployees += 1; employees.Remove(); // delete Employee object currently // referenced by the employees cursor } Debug.Assert(nEmployees == 3); employees.Close(); // close the cursor con.CommitTransaction(); // commit the transaction
In the following example, a simple search is performed to find the employee record with a specified name. A Cursor object to search the name field of the Employee class is instantiated as:
Cursor<Employee> cursor = new Cursor<Employee>(con, Employee.class, "name");
An instance employee is defined and assigned the result of the Cursor class find method (using the name field). The argument for Cursor.find is the desired name value:
Employee employee = new Employee(); employee = cursor.Find("John Smith"); Debug.Assert(employee == null);
Indexes on character and string fields are case-sensitive. In this case, there is no employee record with the name John Smith in the database. The instance employee is returned as NULL. Recalling the previous section Inserting a record in Chapter 2 of this Developer Guide, an employee record for Hyman Farkleburger was created. Therefore, given a search such as:
employee = cursor.Find("Hyman Farkleburger"); Debug.Assert(employee != null && employee.age == 33 && employee.salary == 50000 && employee.companyID == companyID);
The assertion will succeed. Note: assertions are used in this book and the eXtremeDB.Net sample programs in the SDK for the sole purpose of illustrating that eXtremeDB.Net is working correctly. The cursor is terminated:
cursor.Close();
And the transaction is closed. Since it was a ReadOnly transaction, it does not matter whether RollbackTransaction or CommitTransaction is called. We use CommitTransaction by convention, since RollbackTransaction should be reserved for when something has really gone wrong:
con.CommitTransaction();
For further explanation of cursor operations, please refer to the eXtremeDB User Guide.
eXtremeDB .Net User Guide
Simple Iteration
Iterating over all objects of a class is straightforward:
con.StartTransaction(Database.TransactionType.ReadOnly); Cursor<Company> companies = new Cursor<Company>(con, Company.class); foreach (Company c in companies) { // some action... } cursor.Close(); con.CommitTransaction();
Cursor.Search Options
In addition to exact-match search, BTree indexes support GreaterThan, GreaterOrEquals, LessThan and LessOrEquals search operations. To initiate a search, the method startTransaction is called:
con.StartTransaction(Database.TransactionType.ReadOnly);
In the following example, a search is performed to find employee records with a salary greater than a specified value. A cursor instance is created for the salary field of the Employee class:
cursor = new Cursor<Employee>(con, Employee.class, "salary");
The Search method is called with one of the enumerated search types (GreaterThan in this case), and the search value. If the method returns true, then a for loop returns the results in the collating sequence implied by the index.
if (cursor.Search(Cursor.Operation.GreaterThan, 50000)) { foreach (Employee e in cursor) { // some action... } } cursor.Close(); con.CommitTransaction();
In the following example, a search is performed to list employee records, sorted in alphabetical order by last name, then first name. A cursor instance is created for the index named byName" in the Employee class. See the example in the Defining a Database section for the indexes annotation:
cursor = new Cursor<Employee>(con, Employee.class, "byName"); foreach (Employee e in cursor) { Console.WriteLine(e.name); }
Alternatively, the Find method can be called with a search value to find a specific employee by name.
cursor = new Cursor<Employee>(con, Employee.class, "byName"); Employee e = cursor.Find("Norton", "Bob");
Compound indexes can also be used for partial key (prefix) searches. A single element of the index can be specified:
cursor = new Cursor<Employee>(con, Employee.class, "byName");
The Search method is called with one of the enumerated search types (GreaterOrEqual in this case), and the search value. If the method returns true, then a for loop returns the results in the collating sequence implied by the index.
if (cursor.Search(Cursor.Operation.GreaterOrEqual, Norton, )) { foreach (Employee e in cursor) { // some action... } }
Then, in your application code, use the Patricia Trie index as follows:
con.StartTransaction(Database.TransactionType.ReadWrite); // create a cursor for the prefix field of the Operator class Cursor<Operator> cursor = new Cursor<Operator>(con, Operator.class, "prefix"); // launch a prefix match type of search on the Patricia Trie cursor.Search(Cursor.Operation.PrefixMatch, phone); cursor.Close(); con.CommitTransaction();
Please refer to the eXtremeDB User Guide for a complete description of the Patricia Trie index.
RTree Index
To create an RTree index with eXtremeDB.Net, define the field to be indexed as follows:
[Dimension(4)] [Indexable(type=Database.IndexType.RTree)] public float[] coord;
Note that the field type could be any scalar data type, e.g. byte, short, int, long, float, double. To use the RTree index to perform a search:
con.StartTransaction(Database.TransactionType.ReadOnly); // create a cursor for the coord field of the SpatialObject class Cursor<SpatialObject> cursor = new Cursor<SpatialObject>(con, SpatialObject.class, "coord"); // set up the search values float[] rect = new float[4]; rect[0] = rand.NextInt(RADIUS*2) - RADIUS; rect[1] = rand.NextInt(RADIUS*2) - RADIUS; rect[2] = rect[0] + rand.NextInt(MAX_WIDTH); rect[3] = rect[1] + rand.NextInt(MAX_HEIGHT); n = 0; // search the RTree for rectangles that exactly match the // search rectangle. Not that the RTree index declaration // above does not enforce uniqueness, so more than one // rectangle can match the equals condition if (cursor.Search(Cursor.Operation.Equals, rect)) { foreach (SpatialObject obj in cursor) { // do something... n += 1; } }
Search for rectangles in the database that overlap a rectangle provided as a search value:
if (cursor.Search(Cursor.Operation.Overlaps, rect)) {
Search for rectangles that are wholly contained by the coordinates of the search rectangles coordinates (in contrast to an Overlap search):
if (cursor.Search(Cursor.Operation.LessOrEquals, rect)) {
Search for rectangles that wholly contain the coordinates of the search rectangle:
if (cursor.Search(Cursor.Operation.Contains, rect)) {
Or
if (cursor.Search(Cursor.Operation.GreaterOrEquals, rect)) {
Search for rectangles that are wholly contained by the coordinates of the search rectangle, and have no points in common with it:
if (cursor.Search(Cursor.Operation.LessThan, rect)) {
Search for rectangles that wholly contain the coordinates of the search rectangle, and have no points in common with it:
if (cursor.Search(Cursor.Operation.GreaterThan, rect)) {