faircom update guide

FairCom® Update Guide

For

c-tree Plus V7.12

c-tree Server V7.12

c-tree Drivers V3.12

1992-2003 FairCom Corporation

ALL RIGHTS RESERVED

Published by

FairCom Corporation 2100 Forum Blvd., Suite C

Columbia, MO 65203

(573) 445-6833

www.faircom.com

Copyright 1992 - 2003

FairCom Corporation

All rights reserved. No part of this publication may be stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without the prior written permission of FairCom Corporation. Printed in the United States of America.

First Printing, March 2003

c-tree, c-tree Plus, r-tree, d-tree, the c-tree circular disk logo, and FairCom are trademarks of FairCom Corporation.

Macintosh is a trademark licensed to Apple Computer Co.

IBM is a trademark of International Business Machines Corp.

All other trademarks, trade names, company names and product names, contained in this guide are registered trademarks or trademarks of their respective owners.

Specifications are subject to change without notice.

V7.12(0922)(030326)

All Rights Reserved i

Table of Contents

1. OVERVIEW.................................................................................................................... 1 1.1 FairCom Update Guide V7.12 ...................................................................................................... 1 1.2 Partitioned Files ............................................................................................................................ 2 1.3 c-tree Server Cache Subsystem..................................................................................................... 2 1.4 Data Filters and Conditional Index Support.................................................................................. 2 1.5 c-tree Server Features and Enhancements..................................................................................... 3

UNIFRMAT Servers ............................................................................................................... 3 TCP/IP Dead Client Detection................................................................................................. 4 Portability Enhancements ........................................................................................................ 4 Diagnostic and Debug Enhancements ..................................................................................... 4 Other Features and Enhancements........................................................................................... 4

1.6 c-tree Plus API Enhancements ...................................................................................................... 5 1.7 Critical, Serious, and Other Fixes ................................................................................................. 5 1.8 Upgrade and Compatibility ........................................................................................................... 5

Upgrade ................................................................................................................................... 5 Compatibility Notes................................................................................................................. 6 FPUTFGET EXCLUSIVE Files Cached................................................................................. 8

2. PARTITIONED FILES................................................................................................ 11 2.1 Highlights.................................................................................................................................... 11 2.2 Overview..................................................................................................................................... 12 2.3 Implementation ........................................................................................................................... 12

Partition Naming.................................................................................................................... 13 Rules ...................................................................................................................................... 13 Advanced - Maximum Partitions........................................................................................... 13

2.4 Operation..................................................................................................................................... 13 Raw Partition Numbers.......................................................................................................... 14 Unique Keys .......................................................................................................................... 14 Serial Segments (SRLSEG)................................................................................................... 14 Transaction Processing .......................................................................................................... 14 Encryption.............................................................................................................................. 15 Partitioned File Security – File password support ................................................................. 15 Partition Administration Function ......................................................................................... 15

3. C-TREE SERVER CACHE SUB-SYSTEM .............................................................. 17 3.1 Allocating Memory for Data/Index Caches ................................................................................ 17 3.2 Cached Data - Writing to Disk.................................................................................................... 18 3.3 Cache Performance Enhancements ............................................................................................. 19

Advanced Cache Settings ...................................................................................................... 19 3.4 New File-specific Cache Options................................................................................................ 20

Table of Contents

ii Copyright © 2003 FairCom Corporation

Disabling File Cache.............................................................................................................. 20 Dedicated File Cache............................................................................................................. 21 Priming Cache ....................................................................................................................... 22 Avoiding file flush at server shutdown.................................................................................. 23

3.5 Other Notes ................................................................................................................................. 24 SystemConfiguration Enhanced: Cache/Buffer Statistics ..................................................... 24

4. DATA FILTERS AND CONDITIONAL INDEX SUPPORT.................................. 25 4.1 Data Filters .................................................................................................................................. 25 4.2 Variable-length Records with Conditional Expressions ............................................................. 26 4.3 User Defined Conditional Expression Callback Function .......................................................... 27

Expression Parser Improved .................................................................................................. 28

5. C-TREE SERVER FEATURES AND ENHANCEMENTS..................................... 31 5.1 UNIFRMAT Server .................................................................................................................... 31

Limitations............................................................................................................................. 32 Modes .................................................................................................................................... 32

5.2 Dead Client Detection ................................................................................................................. 33 Implementation ...................................................................................................................... 33 Operational Detail.................................................................................................................. 34 Additional Technical Detail................................................................................................... 34

5.3 Portability Enhancements............................................................................................................ 35 Windows XP 64-bit Port Completed ..................................................................................... 35 AIX v5.1 Server Port Completed........................................................................................... 35 Apple MAC Server Unique File Detection............................................................................ 35 MacTCP Dead Client............................................................................................................. 35 Bound Server Shared Library on Linux................................................................................. 35 .NET Compiler Support – Microsoft C/C++ Compiler v13.00 ............................................. 36 UNICODE ICU 2.1 Libraries Updated – Mac OS X Supported ........................................... 36

5.4 Diagnostics and Debug Enhancements ....................................................................................... 36 Locking Activity Enhanced ................................................................................................... 36 New CTIDMP Utility ............................................................................................................ 37 ctflvrfy – New Index Verify Utility....................................................................................... 37 Use sysiocod to Monitor Disk I/O Activity ........................................................................... 37

5.5 Other Server Enhancements ........................................................................................................ 37 Suspend Server Option Enhanced.......................................................................................... 38 New Server Keyword: STARTUP_BLOCK_LOGONS....................................................... 38 Windows Service Shutdown Uses Registry Shutdown Delay............................................... 39 Bound Server Directory Support ........................................................................................... 39 c-tree Server SDK and Bound Server: printf() and exit() Overrides ..................................... 39 PAGE_SIZE Maximum Increased To 64K ........................................................................... 40 Read-only File Mode Improved – Best of Both Ways .......................................................... 40 File Encryption Performance Enhanced ................................................................................ 41 FUNCTION_MONITOR – Log all c-tree Error Numbers .................................................... 41

Table of Contents

All Rights Reserved iii

Launch Executables at Server Launch and Shutdown........................................................... 41 User-defined SIGNAL_READY and SIGNAL_DOWN Functions...................................... 41 Server Idle Flush Cache Keywords Added............................................................................ 42 Block Superfile Member Create/Delete During Dynamic Dump.......................................... 42 Automatic Recovery – Skip Questionable Index Files .......................................................... 43

6. C-TREE PLUS API ENHANCEMENTS................................................................... 45 6.1 Check Lock Status With ctLockRecord ...................................................................................... 45 6.2 FPUTFGET EXCLUSIVE Files Cached .................................................................................... 46 6.3 Special Add Record Feature – “Append Only” mode................................................................. 47 6.4 API Function Enhancements....................................................................................................... 47

Rename Files under Transaction Control .............................................................................. 47 DoBatch Option Reads Just Fixed Portion of Variable-length Record ................................. 48 ctDROPIDX: Permit a non-TRANPROC index file to be dropped ...................................... 48 Windows DLL: c-tree exported functions now have fixed Ordinal Values .......................... 48 New GetCtFileInfo (GETFIL) mode: Return starting offset data records............................. 49 Retrieving the Extended File Creation Block ........................................................................ 49 Frees ISAM Level Locks with LockCtData (LOKREC)....................................................... 49 Allow PutIFileXtd8 Call On Open File ................................................................................. 49 New Function StopUserAsync added – Stop User Asynchronous ........................................ 50 CtreeAsynchronous – New API call added ........................................................................... 50 CtreeFlushFileXtd Added – New Cache Flushing API call .................................................. 50

6.5 m-tree Make System Enhancements ........................................................................................... 51 m-tree’s “custom” Directory Relocated................................................................................. 51 Add Unique Extensions to mtmake Directory Names........................................................... 51 Add Timestamp to mtmake Directory Names ....................................................................... 51

6.6 Other Features and Enhancements .............................................................................................. 52 CTSBLDM Rebuilds Superfile Index Members ................................................................... 52 New Index Segment Mode: Schema-based Serial Segments ................................................ 52 Index Segment Enhanced - Support Segment Offsets > 32767............................................. 52 Prevent False Flags In Fixed Length Data Records............................................................... 53 ctmark Enhanced ................................................................................................................... 53

6.7 New Function Descriptions......................................................................................................... 54 ctFILELIST............................................................................................................................ 55 CtreeAsynchronous................................................................................................................ 57 CtreeFlushFileXtd.................................................................................................................. 58 GetXtdCreateBlock................................................................................................................ 60 PartitionAdmin ...................................................................................................................... 61 SetDataFilter .......................................................................................................................... 64 StopUserAsync ...................................................................................................................... 65

7. C-TREE DRIVER ENHANCEMENTS ..................................................................... 67 7.1 Performance Enhancements ........................................................................................................ 67

Allow overriding number of records reported to SQL engine............................................... 67

Table of Contents

iv Copyright © 2003 FairCom Corporation

7.2 Microsoft Access-Specific Issues................................................................................................ 68 “Invalid index definition” Due to Duplicate Index Names.................................................... 68 #Name? (memory allocation error) Reading With Access 2000 ........................................... 68

7.3 Crystal Reports-Specific Issues................................................................................................... 68 INNER JOIN Support Added ................................................................................................ 68

7.4 Microsoft Visual C++ 6-Specific Issues ..................................................................................... 69 Optionally Disable Floating Point Unit Exceptions .............................................................. 69 Microsoft Visual C++ 6 Query Designer fails to read records with "Catastrophic error" error. ...................................................................................................................................... 69 Microsoft Visual C++ 6 Query Designer fails to open table with "Driver not capable" error when using the ODBC driver in multi-user non-server mode (ODBC and SDK)................. 70

7.5 Driver Fixes................................................................................................................................. 70 Access Violation when Updating Variable-Length Table (ODBC) ...................................... 70 Field Not Found for Key Segment Leads to Access Violation (CR)..................................... 71 Crystal Reports Access Violation "memory cannot be read" Error (CR).............................. 71

8. CRITICAL, SERIOUS, AND OTHER FIXES.......................................................... 73 8.1 Critical Issues .............................................................................................................................. 73

Dynamic Dump extent size overrun ...................................................................................... 73 Bound Server Crash due to Thread Issue............................................................................... 73 Dynamic Dump DSIZ_ERR(443) ......................................................................................... 73

8.2 Serious Issues .............................................................................................................................. 73 Reported 8770 error resolved ................................................................................................ 73 Dynamic Dump Wildcard Recursion Check ......................................................................... 73 Resolved Core Dump During Dynamic Dump Recovery...................................................... 74 Core Dump with Segmented Files / FPUTFGET / ctLOCK_DIRECT................................. 74 8770 Error - High Word Contamination in Calls to Non-Huge Files.................................... 74

8.3 Other Issues................................................................................................................................. 74 Unicode fix: Heterogeneous C/S Unicode support................................................................ 74 Minor Superfile Fixes (0722) ................................................................................................ 74 Dynamic dump recovery BNOD_ERR(69)........................................................................... 74 READ_ERR(36) - Dynamic Dump Recovery Superfile Create............................................ 75 Encrypted Superfile Rebuild Core Dump Resolved .............................................................. 75 Dynamic Dump Wildcard Fix................................................................................................ 75 Dynamic Dump of HUGE files over 4GB DEXT_ERR(599) fixed..................................... 75 CMPIFILX FUSE_ERR(46) and RBLIFILX NSCH_ERR(199) Errors............................... 75 r-tree Client issue resolved .................................................................................................... 76 Custom Server API fix........................................................................................................... 76 Alignment Error fixed – ITIM_ERR(160) and SDAT_ERR(445) ........................................ 76 Dynamic Dump Restore Fix .................................................................................................. 76 Dynamic Dump RCHK_ERR(66) Advanced Encryption Issue ............................................ 76 Dynamic Dump – cancel caused a problem........................................................................... 77 TRANBEG ctAUTOSAVE issue resolved............................................................................ 77 Server - Infinite loop in NEWREC for Superfiles ................................................................. 77

Table of Contents

All Rights Reserved v

Superfile Create error not properly reported.......................................................................... 77 FPUTFGET BIDX_ERR(527) .............................................................................................. 77 Standalone FPUTFGET Huge File SEQ8_ERR(689) Fixed ................................................. 77 LOCLIB Syntax Error Resolved............................................................................................ 78 PRDS_ERR(34) During PreviousKey Call To Server........................................................... 78 AutoRecovery/PermIIndex L65 and terr(8798)errors addressed ........................................ 78 Automatic Recovery/TRANDEP Superfile Member Delete ................................................. 78 Automatic Recovery/Superfiles BSUP_ERR(411) Addressed.............................................. 78 Set Function Target Buffer Vulnerability Resolved .............................................................. 78 OpenISAMContext: Infinite loop with pending dropped indices.......................................... 79 CTRBLEX – Rebuild Example program link resolved ......................................................... 79 Mac OS X 10.1 Compatibility ............................................................................................... 79 EstimateKeySpan and NbrOfKeysInRange V6/V7............................................................... 79 Flush Variable-length Space Management Buffers ............................................................... 79

All Rights Reserved 1

1. Overview 1.1 FairCom Update Guide V7.12

FairCom is pleased to deliver the results of our continuing development efforts for c-tree Plus and the c-tree Server. We hope you will discover that the new features and functionality covered in this update guide deliver significant new benefits to your application development efforts. This update includes the following product versions:

• c-tree Plus V7.12

• c-tree Server V7.12

• c-tree ODBC Driver V3.13

• c-tree Crystal Reports Driver V1.12

This new release marks a major milestone for FairCom. In conjunction with this release of c-tree Plus and the c-tree Server, FairCom is introducing significant new interface technology to make it easier for developers to access the proven core of c-tree Plus. These new interfaces, discussed in separate manuals, complement our traditional ISAM and low-level APIs. While we consider this new interface technology to be solid and have customers using it in production environments, we encourage you to leverage FairCom’s 20+ year history of listening to customer requirements by reporting any specific recommendations on this technology.

This Guide includes enhancements to c-tree Plus, the c-tree Server, and the c-tree Drivers and is made up of the following sections:

Chapter 1. Overview – An overview of this new FairCom release with a brief description of new major features and reference to compatibility notes. (This Chapter)

Chapter 2. Partitioned Files

Chapter 3. c-tree Server Cache Subsystem

Chapter 4. Data Filters and Conditional Index Support

Chapter 5. c-tree Server Features and Enhancements

Chapter 6. c-tree Plus API Enhancements

Chapter 7. c-tree Driver Enhancements

Chapter 8. Critical, Serious, and Other Fixes

The new features and enhancements in this release are outlined below and detailed throughout this manual.

Overview Partitioned Files

2 Copyright © 2003 FairCom Corporation

1.2 Partitioned Files

The c-tree Server now supports a unique feature known as Partitioned Files. A partitioned file logically appears to be one file (or more accurately one data file and its associated index files), but is actually a set of files whose contents are partitioned by the value of the partition key. Both the data files and index files are partitioned. This permits data with a defined range of values for the partition key to be rapidly purged or archived (instead of having to delete record-by-record each record within this range).

An example would be an accounting application wherein if the ‘date’ is the basis of a partition key, then each month’s worth of data would physically be kept in its own file, yet logically be kept in one file with all other months. If the application intends to continually archive 1-year-old data on a monthly basis for example, the process of deleting the 13-month-old data each month would be extremely fast, as c-tree only has to ‘drop’ one physical file from the logical collection with a single function call.

Advantages of partitioned files:

• Maintain data in separate c-tree Plus Data/Index files, while enjoying access from a single host file.

• The partitioned file feature is implemented at the ISAM level with virtually no API changes and is supported by our c-treeDB database layer and other higher-level interfaces into c-tree Plus.

• Use standard c-tree Plus data searches on the entire file or on a member file.

• Easily purge or archive individual member files.

• Available for huge and non-huge Extended files under c-tree Server control.

This feature is documented in detail in Chapter 2.

1.3 c-tree Server Cache Subsystem

The c-tree Server internal data and index file caching schemes provide a balanced approach to minimizing disk activity while ensuring data integrity in environments with high transaction volume.

This release includes c-tree Server cache subsystem performance enhancements and introduces file-specific cache support.

This feature is documented in more detail in Chapter 3.

1.4 Data Filters and Conditional Index Support

This release introduces a new way to customize c-tree’s data retrieval abilities. Like the existing Conditional Index support, the new data filter capability implemented with the SetDataFilter function allows developers to control which data, specified by a conditional statement, is returned to the application.

Overview c-tree Server Features and Enhancements


Filtering lets you specify criteria to temporarily restrict access only to records that meet the filter criteria. Filters are similar to, though less powerful than, queries. c-tree Plus filters in a client/server environment may substantially reduce network traffic since only the records that satisfy the filter condition will be returned. When used in conjunction with c-tree Plus sets, filters provide a powerful way to generate simple queries at the c-tree Plus level.

Unlike the conditional index expressions, which become a permanent part of the index definition (except for temporary indices), a specific filter lasts only until another filter is specified for the same data file or the data file is closed. Further, a specific filter only applies to the calling user: it does not affect the retrieval of records by other users.

This feature is documented in more detail in Chapter 4.

1.5 c-tree Server Features and Enhancements

FairCom has been hard at work adding new functionality to the c-tree Server, including Unified Format (UNIFRMAT) Servers, enhanced dead client detection, portability and diagnostic enhancements, and other general enhancements.

These features are documented in more detail in Chapter 5.

UNIFRMAT Servers

c-tree’s standalone models have supported the concept of Unified Format (UNIFRMAT) for years. In fact, UNIFRMAT is one of c-tree’s most popular features. UNIFRMAT allows machines with alternate architectures related to byte ordering to read and operate on each other’s data.

In the early years of c-tree, the best example of the utilization of this feature was allowing applications in Apple’s Macintosh platform (HIGH_LOW environment) to operate on Windows (LOW_HIGH environment) data.

FairCom now extends this feature to the c-tree Server:

• “HIGH_LOW Host” Server: Operates on LOW_HIGH data (i.e., A c-tree Server on Sun Sparc may read Windows data), but not on native HIGH_LOW data.

• “LOW_HIGH Host” Server: Operates on HIGH_LOW data (i.e., A c-tree Server on Intel Linux server may now read Sun data), but not on native LOW_HIGH data.

This allows an existing system to be replaced with an opposite system without converting the data files.

Overview c-tree Server Features and Enhancements


TCP/IP Dead Client Detection

FairCom has introduced a new method of detecting dead clients, using the standard TCP/IP data stream rather than TCP/IP out of band (OOB) data.

Portability Enhancements FairCom added support for Windows XP 64-bit Edition and AIX 5.1 platforms, extended support for Dead Client Detection and Unique File Detection to the Mac OS 7-9 platform, added Bound Server shared library support for Linux, added .NET compiler support, and updated the Unicode support to ICU 2.0 libraries.

Diagnostic and Debug Enhancements New features were added to aid diagnostic and debugging efforts.

Other Features and Enhancements FairCom has added the following features and enhancements:

• Improved the Suspend Server option to allow the Server to start in Suspended mode and to include internal server threads.

• Enhanced Bound Server directory support and printf()/exit() customization.

• PAGE_SIZE maximum increased from 32 KB to 64 KB.

• Read-only file mode enhanced to allow file sharing.

• Enhanced performance when encryption is in use.

• FUNCTION_MONITOR logs c-tree Plus error values.

• Added SIGNAL_READY and SIGNAL_DOWN support on Windows platforms.

• Server idle cache-flushing keywords added.

• Enhanced automatic recovery – skip questionable indices.

Overview c-tree Plus API Enhancements


1.6 c-tree Plus API Enhancements

FairCom has been hard at work adding new functionality to c-tree Plus, described in Chapter 7, including:

• Added option to check the status of locking on a given record offset.

• Allow EXCLUSIVE file caching in the Standalone Multi-user (FPUTFGET/Peer-to-Peer) operational model

• Added optional ‘Append to end’ capability – Disable deleted space re-use.

• Added option to rename files under transaction control.

• Added a new mode allowing DoBatch to retrieve only the fixed-length portion of targeted variable-length records.

• Allow non-transaction indices to be dropped.

• Windows DLL: c-tree exported functions now have fixed Ordinal Values

• A new GetCtFileInfo mode returns the first byte offset for data records.

• New GetXtdCreateBlock function retrieves extended file creation block.

• A new LockCtData mode frees ISAM-level locks.

• PutIFileXtd8 was adjusted to allow operations on open files.

• New StopUserAsync function allows the client to continue without waiting for the Server to complete the user stop operation.

• New CtreeAsynchronous function added to determine the status of an asynchronous thread or to cancel the thread.

• New CtreeFlushFileXtd function added mode and parm values allow applications to flush cache for specific files and groups of files.

• Various m-tree and other enhancements.

1.7 Critical, Serious, and Other Fixes

Critical issues and other fixes corrected in recent builds are detailed in Chapter 8.

1.8 Upgrade and Compatibility

This section is intended for existing users, outlining basic upgrade requirements and compatibility issues.

Upgrade Most developers will upgrade by:

• Installing the new c-tree Plus as described in the Quick Start Guide.

• Compiling the libraries to match your application.

• Changing the compile scripts or projects to reference the new include/library paths and the new library.

Overview Upgrade and Compatibility


Developers deploying client/server models are advised to upgrade their c-tree Server installations to match the client libraries. While we aren’t aware of any compatibility issues between this release and previous versions, FairCom highly recommends matching the c-tree Plus client library and c-tree Server versions.

c-tree Server users can upgrade as follows:

• Halt all operations, including stopping the Server(s).

• Back up the entire Server directory and any data and index files. If you haven’t done a full backup in a while, this is the best time.

• Remove the Server *.FCS files, except FAIRCOM.FCS. (See “PAGE_SIZE Default Change” in the next section for IMPORTANT information)

• Extract the new Server executable files over the existing files.

• Update the client applications.

• Restart the Server(s) and resume operations.

In addition, developers with applications employing the Standalone Single-user model with Transaction Processing support should ensure they get a clean shutdown of their application, then remove all FCS files associated with transaction processing (L*.FCS, D*.FCS, I*.FCS, and S*.FCS) before installing the new transaction processing based application.

Compatibility Notes Please review the following changes before using this version of c-tree Plus. A review of this complete document is also encouraged.

IMPORTANT - PAGE_SIZE Default Change

As a performance enhancement, the default PAGE_SIZE setting for the c-tree Server was changed from 2048 bytes to 8192 bytes. This requires an adjustment for upgrading Server users and for existing users of superfiles.

First, some background on the server’s page size. Optimizing the page size can minimize the number of disk accesses required to find a key, thereby increasing the throughput of your c-tree Server. The c-tree Server determines the index node size using the PAGE_SIZE keyword when the file is created.

Most modern operating systems use at least a 4096 byte page size (meaning 4096 bytes can be read at one time by the operating system), if not 8192. On any given system, the page size of the operating system is typically the same as the underlying file system’s file allocation block. With matched page sizes, each operating system read retrieves precisely one index node with minimal waste. Some developers report enhanced results with values ranging from one-half of the OS page size up to twice (or higher) the page size. FairCom recommends that your page size be at least as large as your operating system’s page size or a multiple thereof.

There are numerous variables that factor into determining the optimum page size, including the size of the key, the number of keys, the operating system, etc. Generally, applications with heavily used indexes with either a large number of key values, or large key lengths (or both) should provide better throughput with a



PAGE_SIZE equal to, or greater than the file system allocation block. Simple testing with your application will help you determine the correct setting for your precise application.

Users upgrading from an earlier release of the c-tree Server can choose one of the following options:

• Set the PAGE_SIZE setting in the server configuration file to keep the previous default: PAGE_SIZE 2048; or

• The FAIRCOM.FCS file (which stores user, group and security settings) is a superfile. Delete the FAIRCOM.FCS file (losing your User, Group, and File security settings) and allow the c-tree Server to automatically recreate it. Then re-add any User, Group, or File security settings. This resets your ADMIN password to the default, ADMIN, which should also be changed; or

• Compact the FAIRCOM.FCS file using the CTSCMP utility, setting the new_sect_size parameter to the new page size value.

Users of superfiles must either change the PAGE_SIZE setting to match the page size used to create their superfiles (which rely on an exact page size), or use the CTSCMP utility to compact their Standard superfiles. There is no utility to compact Extended superfiles (including huge and segmented files), so Extended superfiles require the correct page size setting. Users of Extended superfiles who want to use a larger page size can change the PAGE_SIZE setting to match the page size used to create their superfiles, export the data to separate files, set the PAGE_SIZE setting to the desired size, and then import the data.

The CTSCMP utility information is repeated here from the c-tree Plus Programmer’s Reference Guide for your convenience:

ctscmp <name of the superfile host to compact> [Y] [new_sect_size]

This utility is designed for the Standalone Single-user model. It compacts and rebuilds a superfile and all of its members provided the required IFIL structures are in place. It creates a compacted version of the file in place, using a temporary file named CTREE.TMP as a destination and then renaming it.

The optional argument Y compacts the file without prompting to confirm.

The optional argument new_sect_size is the sector size of the resulting file.

Change in Link Procedures / Include Paths

When linking applications with this release, keep in mind that the old ctree/custom folder is now created as custom.xxx by ctree.mak, keeping a unique folder for each operational model. Be sure to map existing projects and make files to the new directories. See section 6.5 for more details.



Windows DLL: c-tree functions have fixed Ordinal Values

The m-tree make system creates a DEF file for Windows when dynamic load libraries (DLLs) are defined. Prior to this release, the ordinal values defined for each exported function in this DEF file were not fixed. This has no negative impact on existing installations, but will simplify replacing and sharing DLLs in application linked with this release and future releases. See section 6.5 for more details.

IDX_MEMORY and DAT_MEMORY Default Changes

The Server keywords IDX_MEMORY and DAT_MEMORY, which specify the memory reserved to cache index and data files, now default to 5MB each, providing much better performance than the previous default of 225,000 bytes. This change should have no negative effect on existing users, but is mentioned for informational purposes.

Combining ctREADFIL with ctSHARED

c-tree Plus Client/Server and Bound Server models now support combining the ctREADFIL | ctSHARED file modes. See section 5.5 for more details.

Expression Parser Improved

While adding the new user defined callback routine for conditional indexes, we also made some adjustments to c-tree’s expression code. Now, c-tree assures that all proper alignment considerations are properly handled, and also ensures that the buffer size of any record being evaluated is big enough.

The routines that evaluate the conditional expressions now have access to the fixed length of the data record and the total length of the data record. This permits correct alignment adjustments, and we can detect if insufficient data is available. The latter condition results in a CVAL_ERR(598). The easiest way to produce a CVAL_ERR(598) is to read only the fixed length portion of a data record, but have an expression that relies on fields in the variable length portion of the record.

The CT_STRING and CT_FSTRING fields are now evaluated separately: CT_STRING evaluation fails if the delimiter byte cannot be found; CT_FSTRING uses the defined field length if the delimiter cannot be found.

Because c-tree did not check these conditions in the past, it is possible for users who have misrepresented their data in the DODA to get errors not seen before.

FPUTFGET EXCLUSIVE Files Cached FairCom added the ability to cache files in c-tree’s Multi-user Non-Server (Peer-to-Peer) model (FPUTFGET). FPUTFGET specifically does not cache data. Because it is designed to allow multiple users to manage the data simultaneously, all reads and writes to the database are forced (e.g., all reads are done directly from the disk and write operations flush (sync) all data to disk. The internal acronym FPUTFGET stands for "force put/force get ", or in other words, “force write” and “force-read”.



However, when a file is opened in EXCLUSIVE mode, there is no sharing of the file, and therefore no need to force reads and writes. Cache can be used to gain better performance.

As of this update, when a file is opened in EXCLUSIVE mode in c-tree’s Multi-user Non-Server (Peer-to-Peer) model (FPUTFGET), it will be cached.

NOTE: If this file is not properly closed, its header will be marked with a “dirty” flag, and any subsequent open to this file will fail with a FCRP_ERR(14) error, indicating that the file’s data may not be intact.

In order to take advantage of this feature, use either the InitCTreeXtd or the InitISAMXtd extended initialization call. If a non-extended initialization call is used (INTREE or INTISAM) under FPUTFGET, THERE WILL BE NO DATA CACHE ALLOCATED.

As always, a data file that is NOT opened EXCLUSIVE will not be cached and therefore not use any of the data cache pages.

COMPATIBILITY NOTE: Prior to this change, it was possible to open the file exclusively, and if the file was not properly closed, reopen the file again without error. This is no longer the case, as described above. Prior to this change, all data written to this file would have been forced to disk, therefore no open error was necessary when the file was reopened. Now, because the data is cached, c-tree must indicate an error condition if the file is reopened after not being properly closed.


2. Partitioned Files The c-tree Server now supports a unique feature known as Partitioned Files. A partitioned file logically appears to be one file (or more accurately one data file and its associated index files), but is actually a set of files whose contents are partitioned by the value of the partition key. Both the data files and index files are partitioned. This permits data with a defined range of values for the partition key to be rapidly purged or archived (instead of having to delete record-by-record each record within this range).

hit.dat.001 hit.dat.002 hit.dat.003 hit.dat.* hit.dat.* hit.dat.* hit.dat.* hit.dat.* hit.dat.* hit.dat.010

Host Data File

Partition Rule

Partition Members

hit.dat

• Logical file• Contains no data

• Contains the logic fordistributing partitioned data

• Holds partitioned data• Each file is a unique c-tree data

file that can be accessedindividually or via the host

2.1 Highlights

• Maintain data in separate c-tree Plus Data/Index files, while enjoying access from a single host file.

• The partitioned file feature is implemented at the ISAM level with virtually no API changes and is transparent to our c-treeDB database layer and other higher-level interfaces into c-tree Plus.

• Use standard c-tree Plus data searches on the entire file or on a member file.

• Easily purge or archive individual member files.

• Available for huge and non-huge Extended files under c-tree Server control.

• Requires a special c-tree Server with your partition rule support. Contact FairCom for assistance with generating your custom Server in addition to reviewing Chapter 15, “c-tree Server SDK”, in the c-tree Plus Programmer’s Reference Guide.

Partitioned Files Overview


2.2 Overview

A partitioned file is comprised of a host data file and its associated indices combined with a rule. The rule determines how to map the partition key value (e.g., a date, or invoice number, or some other characteristic) into a particular partition member of the host. The host file does not contain any actual data, but serves as a logical file that appears to contain all the data.

A partition member file has the same definition as the host, but only holds data whose partition key maps to the member. For example, customer orders may be partitioned by calendar quarter. All orders booked in the same quarter are stored in the same member. A member is comprised of a standard c-tree data file and its associated indices.

To add data to the partitioned file, simply call an add routine for the host. The code will add the record in the proper partition member, creating the partition member if necessary. Rewriting a data record may move the record between partitions if the partition key entry for the record is changed. Under transaction control, such a delete/add record operation is done atomically.

Searching the partitioned file in key order is fairly straightforward and efficient if the key is the partition key (the key used to determine which partition should contain the record). Searches on other, non-partition, keys are less efficient than normal because the record may exist in any of the partition members. The more active partition members there are in the logical file, the less efficient the non-partition key search will be.

It is possible to manage the partition members so that archiving or purging partitions is very quick and efficient.

2.3 Implementation

Implementing partitioned files requires a c-tree Plus client-side library with partition capabilities and Extended files with the ctPARTAUTO extended file mode. To implement partitioned files:

1. Activate the partitioned files capabilities at compile time with the ctPARTITION define. This define is on by default, provided the ctHUGEFILE, RESOURCE, CTS_ISAM, and ctCONDIDX defines are in place. Check ctree.mak/ctoptn.h and ctopt2.h for the current settings.

2. To create a partitioned file (using the default partition naming, partition rule, and maximum number of partitions), simply create an Extended file with ctPARTAUTO in the x8mode parameter of the extended file creation block. Partitioned files do not have to be HUGE unless the logical file size will exceed the 2GB/4GB limit, but they do require the extended header, i.e., they are Extended files; hence they cannot be accessed by V6 code even when the individual partition file is accessed separately.

Partitioned Files Operation


Partition Naming The partition file name will be the base file name with the 3-digit raw partition number as the file extension. Only automatic naming is available at this time. See the function partnam() in ctpart.c for the current naming algorithm.

Rules The partition key can be set when the file is created using the new prtkey parameter of the extended file creation block. This value defaults to 0, indicating the first key associated with the data file. Set this value to the relative key number for the desired index if the default is not appropriate for your application.

The current default rule, developed for testing purposes only, uses a simple algorithm to divide added records into partitions based on the first byte of the selected key as a test of the partition capability. This option is currently hard-coded into the Server and is still under development. See the function kprawno() in ctpart.c for the sample algorithm. At this time, only the skeleton support has been added. No actual rule processing has been encoded.

Advanced - Maximum Partitions By default, 16 bits of the 8-byte record offset are used to reference the raw partition number, allowing each partitioned file to support up to 65535 member files over its lifetime. This can be adjusted using the callparm parameter of the extended file creation block, where a value of 0 defaults to 16, values less than 4 default to 4 (maximum 15 member files), and 32 is the maximum value (4,294,967,295 member files).

2.4 Operation

For now, we make two major assumptions: 1) The c-tree Server assigns data records to a partition by applying the data file’s

partition rule to the partition key; and 2) Partitions are assigned in increasing order of the partition key values. That is, if

KeyValue2 > KeyValue1, then the partition assigned to KeyValue2 will be the same as or after the partition assigned to KeyValue1.

Neither of these two assumptions is absolutely critical, but the second assumption does permit much more efficient key searches when the relationship between key values and partitions is well ordered.

Once the host file is created, the operation of partitioned files should be invisible to the application. The functions that add, update, and delete records are the same for both partitioned and non-partitioned files. However, functions requiring a record offset must use the ctSETHGH and ctGETHGH functions, even if the partitioned files are not HUGE to ensure the high-order bytes are included, as described below.



Raw Partition Numbers The partition numbers are stored in the higher order bytes of the 64-bit record offset. This allows the ISAM API calls to remain unchanged. Simply change the parameters of your file creation call, and your application is ready to use partitioned files.

The callparm parameter of the extended file creation block is used to determine the number of bits reserved for the raw partition number. The number of bits determines the total number of raw partitions for the entire life of the host file. This is not the number of partitions active at one time. Raw partitions are not reassigned. A value of zero defaults to 16 bits for the partition number. Any other value less than 4 is set to 4. This implies support for a total of only 16 partitions. The maximum number of bits for the partition number is 32.

The raw partition numbers must be 1 or greater. When passing a file position that includes a partition number to a routine, the partition number is encoded in the high order bits of the high order word. Ordinarily, the application will only get such information from a call to CurrentFileOffset followed by a call to ctGETHGH.

Use the PartitionAdmin function to increase or decrease the lowest permitted partition number, called the “base” partition number. The system enforces an absolute lowest value for the base of one (1), but PartitionAdmin can be used to change the base as long as it is one or greater. However, when changing this base value, PartitionAdmin must ensure that no inconsistencies will arise. One cannot increase the base value if it would eliminate any active or archived partitions (however it can eliminate purged partitions).

Unique Keys

Unique indices are managed using the host index files to maintain values across all partitions for unique non-partition keys. The values in the host index use their partition number (plus 1) as their associated values. The partition index contains the key value with the associated record position within the data partition. This permits the partitioned files to be treated as a self-contained set of ISAM files, and still enforces unique keys across all partitions. Of course, there is a performance penalty associated with indexing the non-partition unique keys twice.

A new duplicate key flag value, 2, permits unique keys within a partition, but no check is made for global uniqueness across all partitions.

Serial Segments (SRLSEG) SRLSEG is managed by using the data file host header to maintain the serial numbers used by key segments across all partitions. SRLSEG key segments can be used to make a key value unique, and this still works with partitioned files.

Transaction Processing A partitioned file that supports transaction processing must be a transaction-dependent (TRANDEP) file. This is automatically enforced.



Encryption Encryption is not currently supported with partitioned files.

Partitioned File Security – File password support File passwords are supported for partitioned files. A partition created on the fly is assigned the same security information as the host file.

Partition Administration Function The Partition Administration function, PartitionAdmin, allows on-the-fly adjustment to the partitions associated with a given host file. This includes the capability to:

• Add, remove, or archive partition(s)

• Modify lower limit of the raw partition number

• Modify limit on the number of partitions

• Reuse the raw partition number of a purged member

• Activate archived member(s)

• Return a member file status For additional information on PartitionAdmin, see the function description in Section 6.7, “New Function Descriptions”.


3. c-tree Server Cache Sub-System

The c-tree Server internal data and index file caching schemes provide a balanced approach to minimizing disk activity while ensuring data integrity in environments with high transaction volume. Data integrity, transaction control, and concurrency issues are but a few of the many internal complexities involved in the cache sub-system. In order to properly put into perspective the benefits of the caching sub-system, one must understand all the options, facts and trade-offs involved in supporting an advanced cache in environments with high transaction volumes.

This section clarifies questions raised about the c-tree Server cache sub-system and describes features affecting cache management:

• Changing the number of operations performed in one pass of the cache

• Adjusting the amount of a large record to keep cached.

• Disabling cache for a file.

• Dedicating cache to a file and limiting dedicated cache.

• Loading files to cache at open.

• Avoiding file flush at server shutdown.

3.1 Allocating Memory for Data/Index Caches

In general, c-tree Server cache is easy to configure. By placing the following keywords in the c-tree Server configuration, the Server Administrator defines the memory size for both data and index caches. The values represent bytes. This example sets each cache to 20MB:

IDX_MEMORY 20000000 DAT_MEMORY 20000000

Because the default value is very low (250,000 bytes), it is strongly recommended to specify an increased cache size. The c-tree Server keeps the most recently accessed data in memory using a “most-recently-used” scheme to determine which pages remain in memory and which do not.

Prior to this release, setting IDX_MEMORY or DAT_MEMORY to very large values (e.g., over 100MB) could degrade performance. While the cache hashing mechanism is very efficient for random access, certain operations (transaction checkpoints and file closure) required exhaustive inspection of each cache page. The larger the cache, the more overhead this entailed. Thus, at a certain point, the size of the cache actually began to add overhead as opposed to more efficiency.

This release adds cache-management techniques allowing the server administrator to set the cache sizes as high as reasonably possible, without defeating the purpose, as well as options such as memory resident files and priming the cache for a file. These new features are described in detail in sections 3.3-3.5 below.

c-tree Server Cache Sub-System Cached Data - Writing to Disk


3.2 Cached Data - Writing to Disk

In general, the purpose of the cache is to prevent reading and writing data to/from the disk. The data managed in the cache accesses the disk for a number of reasons:

File Extension Size: If any index or data file has been defined to have a “file extension size” (as defined in the IFIL or IIDX structures), these files are defined to require disk writes. Specifying an extension size tells c-tree Plus to extend the file by the given number of bytes each time it runs out of space. To secure this extension, c-tree Plus forces the disk write, thus ensuring the operating systems file allocation table (FAT) registers this new extended size.

The best way to ensure maximum performance is to define a ZERO (0) FILE EXTENSION SIZE for all files. A value of zero suppresses this operating system overhead except for transaction-processed files.

File Header Updates: There are a number of situations where the server updates the control header portion of a c-tree file. Updating the number of active records in the file is a good example. Typically a file header update does not necessitate a disk write, but some situations cause a required disk write. Perhaps the best example is when adding a new root node to an index file, which requires a file header write. These types of circumstances are very infrequent and should offer no apparent overhead to the overall performance of the system.

Not Enough Cache Space: Of course, if the demand on the cache exceeds its defined limit, data not currently being used will be written to disk. Cache pages are analyzed using a “most-recently-used” scheme to determine which pages remain in cache.

Middle portion of Variable Length Records: c-tree does not store the middle portion of variable length records in cache, only the first and last pages. This prevents large blocks of data from consuming the cache and also alleviates the management of a large number of cache pages for any one particular record. There is increased overhead when managing a large volume of individual cache pages for any one particular record. The cache pages for consecutive segments of a record (where a segment fills a cache page) are completely independent of each other - they are not stored in consecutive memory. I/O is performed separately for each cache page, therefore if a large number of cache pages exist; it is likely to be slower than faster.

File Open and Close Operations: Any action to open or close a file causes disk I/O. Besides the obvious reasons, a number of internal operations take place during these critical operations.

Transaction Control: To manage transactions, there are a number of transaction processing operations that require disk flush activity. Securing transaction logs, issuing checkpoints, and handling transaction cache ageing are a few examples of disk activity that support automatic recovery to ensure data integrity.

Write-Thru Cache? FairCom does not categorize its cache as what some have called a “write-thru” cache. Perhaps the file size extension issue discussed above has caused this effect. Given a zero file size extension, all “adds” to the database should go

c-tree Server Cache Sub-System Cache Performance Enhancements


directly to cache, with no disk activity. Our goal is to offer the most efficient caching scheme while assuring complete data integrity.

Does c-tree cache everything? Not everything is cached, for example:

a) As mentioned, the middle portions of variable length records are not cached.

b) As defined below, a file may be specified “not to be cached”.

c) Data “not recently used” when the cache is full will generate disk activity.

When writing/reading a 100K record, do you try to cache it, thus wiping out the utility of the cache? No! Because we only retain the first and last page of a variable length record in the cache, this problem does not exist.

Are there any size limits on what you try to cache? The approach used for variable length records alleviates any concern for size limits on what we “try to cache”.

3.3 Cache Performance Enhancements

The previous buffer/cache management scheme focused on random access. Regardless of the size of the buffer/cache space, the Server could access individual pages very efficiently through a simple hashing mechanism. However, to find each page that is updated, during checkpoint processing for example, required an exhaustive check of every page.

The new behavior augments the hashing mechanism with linked lists of buffers so that only those buffers of interest need to be processed. Buffer and cache pages are placed on linked lists by file so file close and flush operations do not need to search the entire buffer/cache space.

Advanced Cache Settings While the default cache settings should be considered efficient, it is possible to adjust the following settings to further enhance performance.

Buffer Run Length

The new Server keyword, BUFFER_RUNLENGTH, which defaults to 10, determines how many consecutive write operations are performed while walking a list of buffer/cache pages before the mutex controlling access to the list is released and then reacquired. Releasing the mutex permits other threads to acquire control of the list. If this value is set to a negative value, it is ignored.

For bound server applications, the brunlen member of the ctINIT structure is used to override the default.

Multi-page cache

The c-tree data cache uses the following approach to cache data record images:

• If the data record fits entirely within one or two cache pages (PAGE_SIZE bytes per cache page), then the entire record is stored in the cache.

c-tree Server Cache Sub-System New File-specific Cache Options


• If the data record image covers more than two cache pages, then the first and last segments of the record are store in the cache, but the middle portion is read from or written to the disk. These direct I/O’s are efficient operations since they are aligned and are for a multiple of the cache page size.

The nature of this approach can now be modified. Set the new Server keyword MPAGE_CACHE to a value, N, greater than zero, and records that fall within N+2 cache pages will be stored entirely within the cache. The default value is zero, which causes the cache to behave as described above.

Note: Setting MPAGE_CACHE greater than zero does NOT ensure faster system operation. It is more likely to be slower than faster. It does cause more of a record to be in cache, but there is increased overhead managing each individual cache page. The cache pages for consecutive segments of a record (where a segment fills a cache page) are completely independent of each other. They are not stored in consecutive memory and I/O is performed separately for each cache page. This configuration option should only be used for special circumstances with careful, realistic testing to verify any performance enhancement.

Note: Even a record smaller than a single cache page may require two cache pages because the record position is generally not aligned on a cache page boundary.

For a bound server, the MPAGE_CACHE configuration parameter is stored in the mpagche member of the ctINIT structure. For Standalone applications, the configuration parameter is stored in the “global” ctmpagcache integer variable. When ctNOGLOBALS is defined, the application instance must be registered before ctmpagcache can be reset. A client cannot affect this value on the Server.

3.4 New File-specific Cache Options

The c-tree Server can now be instructed to assign cache space to a given file and to prime, or pre-load, a file into cache. To make a data file “memory-resident”, use both the PRIME_CACHE and SPECIAL_CACHE_FILE keywords to load dedicated cache pages at file open.

Disabling File Cache

In some cases, it might be beneficial to define that a certain file NOT be cached.

For example, if a file contains very large variable length records (BLOBS), it might be more efficient to bypass the cache and rely solely on the operating system’s cache support. The c-tree Server does not store the full variable length record in cache, but retains the first and last page of the variable length record. This prevents large blocks of data from consuming the cache and also alleviates the management of a large number of cache pages for any one particular record.

To disable cache for a given file, use the following server configuration keyword:

NO_CACHE <data file name>



Note: <data file name> may include a wildcard pattern using ‘?’ for a single character and ‘*’ for zero or more characters. The Server Administrator can specify multiple NO_CACHE configuration entries.

Caching can only be turned off for entire superfiles (i.e., the superfile host), not for individual superfile members. Index files require the use of index buffer pages and must be cached.

For Bound Server or Standalone Single-user models, use the ctFILELIST call below, where filename points to the file name and bytes is a LONG holding the number of bytes of dedicated cache to assign to the file. ctFILELIST must be called prior to the creation or opening of the file for which the no cache is to be assigned.

ctFILELIST(filename,&bytes,NULL,ctNO_CACHElist,ctADDfilelist);

Dedicated File Cache

The SPECIAL_CACHE_FILE keyword dedicates a specified amount of cache memory to a particular Extended data file. This allows the Server Administrator to specify files that are critical to maintain in cache memory at the expense of the “least-recently-used” (LRU) approach, where a new cache page replaces the LRU existing page.

The server configuration file specifies the amount of dedicated cache for specific data files with a configuration file entry in the form:

SPECIAL_CACHE_FILE <datafilename>#<bytes: dedicated cache>

For example, the following keywords specify 100,000 bytes of dedicated cache for customer.dat and 400,000 bytes for data\inventory.dat:

SPECIAL_CACHE_FILE customer.dat#100000 SPECIAL_CACHE_FILE data\inventory.dat#400000

NOTE: The special cache only applies to data files, and until a file is physically closed or until UpdateHeader is used as described below, the special cache pages stagnate: once they hold a particular page of the file (i.e., a chunk of the file at a specific file offset), they always hold the same page. Use the UpdateHeader function to:

• Stop using dedicated cache for a file. UpdateHeader(datno,0,ctSPLSTOPhdr);

• Refresh the dedicated cache pages so that they may be assigned to different pages of the file. UpdateHeader(datno,0,ctSPLREFRESHhdr);

• Restart the use of dedicated pages after a call to stop their use. UpdateHeader(datno,0,ctSPLRESUMEhdr);

The second parameter in UpdateHeader is ignored in these three cases.

For Bound Server or Standalone Single-user models, use the ctFILELIST call below, where filename points to the file name and bytes is a LONG holding the number of bytes of dedicated cache to assign to the file.:



ctFILELIST(filename,&bytes,NULL,ctSPLCACHElist,ctADDfilelist);

ctFILELIST must be called prior to the creation or opening of the file for which the special cache is to be assigned.

Limiting Special Cache Space

To specify the percentage of the overall data cache space that may be dedicated to individual files, use a configuration entry of the form:

SPECIAL_CACHE_PERCENT <percentage>

For example, the following keyword would permit up to 10% of the total data cache pages to be assigned to files on the special cache file list:

SPECIAL_CACHE_PERCENT 10

To disable any special cache, enter –1 for the percentage. The percentage defaults to 50% and the maximum amount that can be specified with the keyword is 90%.

For a bound server, the dspllmt member of the ctINIT structure may be used to override the 50% default. For a single-user, standalone application, the ct_dxspllmt must be set to the percentage before the system is initialized (which may require a call to RegisterCtree if ctNOGLOBALS is defined).

Priming Cache

The c-tree Server optionally maintains a list of data files and the number of bytes of data cache to be primed at file open. When priming cache, the Server reads the specified number of bytes for the given data file into data cache when physically opening the data file.

Data files are added to the priming list with configuration entries of the form:

PRIME_CACHE <data file name>#<bytes primed>

For example, the following keyword instructs the Server to read the first 100,000 bytes of data records from customer.dat into the data cache at file open:

PRIME_CACHE customer.dat#100000

A dedicated thread performs cache priming, permitting the file open call to return without waiting for the priming to be accomplished.

Use PRIME_CACHE with the SPECIAL_CACHE_FILE keyword to load dedicated cache pages at file open.

To prime index files, use configuration entries of the form:

PRIME_INDEX <index file name>#<bytes primed>[#<member no>]

If the optional <member no> is omitted, all index members are primed. If an index member number is specified, only that member is primed.

For example, the following keyword instructs the Server to read the first 100,000 bytes of index nodes in customer.idx into the index buffer space at file open:

PRIME_INDEX customer.idx#100000



The nodes are read first for the host index, and then for each member until the entire index is primed or the specified number of bytes has been primed.

The following example restricts the priming to the first member (the index after the host index):

PRIME_INDEX customer.idx#100000#1

The <data file name> or <index file name> can be a wild card specification using a ‘?’ for a single character and a ‘*’ for zero or more characters.

To use the prime cache file list for a bound server, use ctFILELIST calls of the following form, where filename points to the file name, bytes is a LONG holding the number of bytes of index buffers to prime at file open, and member is a LONG holding the index member number. A negative value for member indicates the entire index.

ctFILELIST(filename,&bytes,NULL,ctPRICACHElist,ctADDfilelist); ctFILELIST(filename,&bytes,&member,ctPRIINDEXlist,ctADDfilelist);

This function must be called prior to the opening of the file for which the cache is to be primed. The cache priming is performed by a dedicated thread launched by the Server at the end of the file open. This permits the open to return without waiting for the priming to be accomplished.

Avoiding file flush at server shutdown With the recent enhancements to the c-tree Server cache system, it is possible for users to have extremely large cache sizes (e.g., 2GB). With caches this large, it may be possible for a file to never be written to disk during its entire life cycle. When the c-tree Server is shut down, it begins to flush files to disk. In the case of a “scratch” or “temp” file, the application vendor may not care if c-tree flushes this file to disk. Understandably, if the file is large, this will cause extra time for the c-tree Server to shut down, and you might not want to suffer this extra shutdown time.

In order to handle this type of situation, the new server keyword NO_SHUTDOWN_FLUSH has been added for when it may be advantageous to skip file flushing during server shutdown. Non-transaction controlled files can be specified as shown below for this treatment. Such a file will be corrupted after shutdown (if file flushing was indeed skipped). For a server, configuration entries are of the form:

NO_SHUTDOWN_FLUSH <file name>

Note that <file name> may specify a wildcard pattern.

To use the skip file flush at shutdown file list for a bound server, use calls of the following form, where filname points to the file name, bytes and member are LONG, but the values are ignored:

ctFILELIST(filname,&bytes,&member,ctNO_FLUSHlist,ctADDfilelist);

c-tree Server Cache Sub-System Other Notes


3.5 Other Notes

• The ctFILELIST processing, used to store and retrieve cache parameters, ignores the mirrored portion of a file name. Only the primary file is cached.

• If multi-byte file names are supported (e.g., Unicode), the configuration information (or direct call to ctFILELIST) must have the file names in UTF8 form (i.e., a NULL terminated string).

SystemConfiguration Enhanced: Cache/Buffer Statistics It is now possible to capture system-wide cache and buffer statistics. Cache pages hold data record images, and buffers hold index nodes. This allows your application to track the use of these resources.

SystemConfiguration now returns nine new values referenced with the following constants used as subscripts in the output array of LONG values:

cfgCACHE_PAGES Available cache pages cfgCACHE_PAGES_INUSE Cache pages in use cfgCACHE_PAGES_MXUSE Maximum cache pages used cfgCACHE_PAGES_DED Available dedicated cache pages cfgCACHE_PAGES_DEDINUSE Dedicated cache pages in use cfgCACHE_PAGES_DEDMXUSE Maximum dedicated cache pages used cfgBUFFER_PAGES Available index buffers cfgBUFFER_PAGES_INUSE Index buffers in use cfgBUFFER_PAGES_MXUSE Maximum index buffers used

NOTE: The dedicated cache pages are a subset of the regular cache pages.

Please see the following example for the use of this feature.

LONG ctcfg[ctCFGLMT]; SystemConfiguration(ctcfg); printf(“\nAvailable Dedicated Cache Pages – %ld“, ctcfg[cfgCACHE_PAGES_DED]);


4. Data Filters and Conditional Index Support Developers consistently search for more efficient ways to retrieve data. While it is relatively easy to acquire huge masses of data, the more data available, the more critical it becomes to quickly and efficiently retrieve the specific data needed this instant.

A variety of methods have been implemented within c-tree Plus to deal with this issue: sorted indices, batch processing, set functions, conditional indices, and now data filters.

This chapter introduces the new data filter capability and new enhancements to the existing conditional index support.

4.1 Data Filters

Like the existing Conditional Index support, the new data filter capability implemented with the SetDataFilter function allows developers to access a limited portion of the data specified by a conditional statement. Unlike the conditional index expressions, which become a permanent part of the index definition (except for temporary indices), a specific filter lasts only until another filter is specified for the same data file or the data file is closed. Further, a specific filter only applies to the calling user: it does not affect the retrieval of records by other users.

Filtering lets you specify a criteria to temporarily restrict access only to records that meet the filter criteria. Filters are similar to, though less powerful than, queries. c-tree filters in client/server environment may substantially reduce network traffic since only the records that satisfy the filter condition will be returned. When used in conjunction with c-tree sets, filters provide a powerful way to generate simple queries at c-tree Plus level.

ISAM level record requests, including sets and batches, skip over records failing the filter transparently to the user. In client/server, this may substantially reduce network traffic since only the records that satisfy the filter will be returned. For example, with a data file of 1,000 records, if a filter excludes 900 of the records, a FirstRecord-NextRecord loop results in only 100 calls to the Server, not 1,000 calls. Generally, an index can also be used to help reduce unnecessary record retrievals, but the filter allows for ad hoc retrievals.

For partitioned files, call SetDataFilter for the host data file and the filter automatically applies to each partition member file.

The #define ctFeatDATFLT supports data file filtering on record reads. By default, ctFeatDATFLT is activated with ctCONDIDX.

See the SetDataFilter function description for additional details.

Data Filters and Conditional Index Support Variable-length Records with Conditional Expressions


4.2 Variable-length Records with Conditional Expressions

When using data filters or conditional indices with variable length data files, a record retrieval that does not bring back the entire record may return the error CVAL_ERR(598), which indicates the expression could not be evaluated.

There are two types of retrievals that result in less than the entire record being read:

1) Calling the fixed length versions of the ISAM routines such as FirstRecord or NextRecord instead of FirstVRecord or NextVRecord. This will not cause the CVAL_ERR(598) if the expression only requires information in the fixed length region of the record.

2) Calling the variable length versions of the ISAM routines with a buffer length insufficient to hold the entire record. This will likewise not cause a problem if the amount read contains all the fields used in the expression.

When an ISAM call fails (with a CVAL_ERR(598) or some other error), the current ISAM position is NOT updated. Therefore the following pseudo-code sequence will NOT work because the FirstRecord did not establish the failing record as the current ISAM position, and the GETVLEN() call would reference the record at the current ISAM position before the FirstRecord call:

SETFLTR(datno,…); if (FRSREC(…) == CVAL_ERR) { vlen = GETVLEN(datno); rc = FRSVREC(…,&vlen); }

Using the variable length versions of the ISAM routines provides a workable approach. The following pseudo-code works, with one proviso – the subsequent calls to the ISAM routine can also fail with a CVAL_ERR(598)because they may have skipped forward to an even larger record:

SETFLTR(datno,…); oldlen = vlen; if (FRSVREC(…,bufr,&vlen) == CVAL_ERR && oldlen < vlen) { free(bufr); oldlen = vlen; bufr = calloc(vlen); rc = FRSVREC(…,bufr,&vlen); }

The second call to FirstVRecord could also return the CVAL_ERR(598)because while the record that originally caused the CVAL_ERR(598)can now be read completely, if it failed the filter, the next records will be read automatically until a record is found that passes the filter; but these subsequent reads can also encounter a record that is bigger than the new buffer size.

The following pseudo-code loop should work with any of the variable length versions of the ISAM calls:

Data Filters and Conditional Index Support User Defined Conditional Expression Callback Function


SETFLTR(datno,…); oldlen = vlen; while (xyzVREC(…,bufr,&vlen) == CVAL_ERR && oldlen < vlen) { free(bufr); oldlen = vlen; bufr = calloc(vlen); } if (isam_err) then problem or no record found else success If one knows ahead of time that there is a maximum record length for the file, then simply using a buffer of this know maximum size eliminates the need to loop over the CVAL_ERR(598) caused by an insufficient buffer size.

4.3 User Defined Conditional Expression Callback Function

c-tree currently supports the ability to create a “Conditional Index” by which the developer may supply an expression which is used to “condition” or “qualify” the entries made in this index. Whenever an “ADD or UPDATE” is performed to the data records of the file, c-tree’s internal expression evaluator evaluates the conditional index expression to properly maintain this special index.

c-tree now allows the developer to call a developer-defined callback function to perform the evaluation. Instead of calling c-tree’s internal expression evaluator to analyze the criteria, the user-defined function is called, allowing the developer to control the criteria in his/her own code. This gives the developer absolute control over conditional indexes.

The module ctclbk.c contains the routines declared below. ctfiltercb is called when a user-defined filter must be evaluated. Whenever a file open retrieves stored conditional index callback expressions, when a conditional index callback expression is created, or when a SetDataFilter callback expression is created, the ctfiltercb_init routine is called with a pointer to the callback expression. Whenever a file is closed, or a SetDataFilter is cleared, the ctfiltercb_uninit routine is called for each callback expression.

NINT ctfiltercb(pTEXT Clbk, pVOID Recptr, pConvMap Schema, VRLEN fixlen, VRLEN datlen pinHan)

NINT ctfiltercb_init(pTEXT Clbk pinHan) NINT ctfiltercb_uninit(pTEXT Clbk pinHan)

Parameter Description pTEXT Clbk Pointer to a NULL terminated ASCII string beginning with

the ctCNDXclbkCHR (@) character. Presumably, the string starting in the 2nd position (i.e., Cblk + 1) points to a callback function designator.

pVOID Recptr Pointer to a record image pConvMap Schema Pointer to a record schema



Parameter Description VRLEN fixlen Fixed length of the record image VRLEN datlen Full length of the record image pinHan A macro the converts to ctWNGV for standalone or client

libraries defining ctNOGLOBALS or lctgv for Server or Bound Server code.

Stub functions with a simple debug print statement are part of the distribution. A developer taking advantage of the conditional expression callback routine must adapt the callback functions. As described above, Clbk points to an ASCII string beginning with the at sign (@). The string following the @ is completely arbitrary, and must be interpreted to determine what type of callback routine is intended, assuming more than one type of callback is required by the application. A simple scheme would use callback strings with a unique character in the second position (just following the @). This would permit a simple switch statement to route the callback to the desired code, as shown in the following pseudo-code:

/* ** my callback strings are: @CustomerNumber ** @ZeroBalance ** @TotalFunds */ switch (*(Clbk + 1)) { case ‘C’: do the Customer Number check break; case ‘Z’: do the Zero Balance check break; case ‘T’: do the Total Funds check break; default: set the return code to –CVAL_ERR to indicate the filter could not be evaluated break; }

In this type of scheme, the only significant portion of the callback designator is the first character after the @.

ctfiltercb_init is expected to return zero (0) on error and non-zero on init, though at this point the error return is ignored. ctfiltercb_init should set a state variable so the ctfiltercb_uninit knows whether the init was called for this particular callback filter.

ctfiltercb_uninit is expected to return zero (0) on error and non-zero on uninit, though at this point the error return is ignored. ctfiltercb_uninit should check a state variable to determine whether or not ctfiltercb_init was called for this particular callback filter.

Expression Parser Improved Compatibility Note: While adding the new user defined callback routine for conditional indexes, we also made some adjustments to c-tree’s expression code.



Now, c-tree assures that all proper alignment considerations are properly handled, and also ensures that the buffer size of any record being evaluated is big enough.

The routines that evaluate the conditional expressions now have access to the fixed length of the data record and the total length of the data record. This permits correct alignment adjustments, and we can detect if insufficient data is available. The latter condition results in a CVAL_ERR(598). The easiest way to produce a CVAL_ERR(598) is to read only the fixed length portion of a data record, but have an expression that relies on fields in the variable length portion of the record.

The CT_STRING and CT_FSTRING fields are now evaluated separately: CT_STRING evaluation fails if the delimiter byte cannot be found; CT_FSTRING uses the defined field length if the delimiter cannot be found.

Because c-tree did not check these conditions in the past, it is possible for users who have misrepresented their data in the DODA to get errors not seen before.


5. c-tree Server Features and Enhancements FairCom has added new functionality to the c-tree Server, including Unified Format (UNIFRMAT) Servers, enhanced dead client detection, portability and diagnostic enhancements, and other general enhancements.

5.1 UNIFRMAT Server

Data is stored high/low on disk

Data files are interchangeable

WindowsLow/High

SGIHigh/Low

SunHigh/Low

LinuxLow/High

WindowsLow/High

SGIHigh/Low

SunHigh/Low

SunHigh/Low

Clients

c-tree Servers

Data

High/LowData

High/LowData

UniformatServer(High/Low)

StandardServer

Figure 5-1

c-tree’s standalone models have supported the concept of UNIFRMAT for years. In fact, UNIFRMAT is one of c-tree’s most popular features. UNIFRMAT allows machines with alternate architectures related to byte ordering to read and operate on each other’s data.

In the early years of c-tree, the best example of the utilization of this feature was allowing applications in Apple’s Macintosh platform (HIGH_LOW environment) to operate on Windows (LOW_HIGH environment) data.

FairCom now extends this feature to the c-tree Server:

• “HIGH_LOW Host” Server: Operates on LOW_HIGH data (i.e., a c-tree Server on Sun Sparc may read Windows data), but not on native HIGH_LOW data.

• “LOW_HIGH Host” Server: Operates on HIGH_LOW data (i.e., aa c-tree Server on Intel Linux may read Sun data), but not on native LOW_HIGH data.

c-tree Server Features and Enhancements UNIFRMAT Server


This allows an existing system to be replaced with an opposite system without converting the data files. For example, a standard c-tree Server on a Sun Sparc system (HIGH_LOW) could be replaced with a LOW_HIGH Host c-tree Server on a Linux (Intel) system by simply installing the new Linux c-tree Server software and copying the existing data to the Linux system as shown in Figure 5-1.

Limitations As with UNIFRMAT for the standalone models, support is only available for non-transaction processed files. The client application will receive error FTYP_ERR(53) if it attempts to access a TRNLOG, PREIMG, or LOGIDX file.

The c-tree Server requires a specially generated executable to support UNIFRMAT. You must also relink client applications with the V7.12 libraries. Older clients will return a CUNF_ERR(735) error when connecting to a UNFRMAT Server.

A special UNIFRMAT Server can only access files of the opposite flavor to the operating system. If a client attempts to access a native-format file, the Server returns error HDR8_ERR(672). For example, a UNIFRMAT Server for an Intel-based Linux machine (native LOW_HIGH format) can only access HIGH_LOW files.

A DODA is required for automatic data translation and index management if key segments include binary fields. See “Automatic Mode” below.

Modes There are two modes of UNIFRMAT available: Automatic and Manual. FairCom recommends using a DODA to implement Automatic mode.

Automatic Mode

This is the default mode when using the extended initialization routines, such as InitISAMXtd. With a UNIFRMAT Server, the byte flipping of all binary fields managed by c-tree Plus and all of the binary fields in your data records is automatically handled by c-tree Plus.

In order for c-tree Plus to properly manipulate your binary fields, a dynamic object definition array (DODA) must be defined and a PutDODA call must be made after the file is created.

Manual Mode

In this mode only the binary fields managed by c-tree Plus are transposed. The byte flipping of any binary fields stored in your data records is YOUR RESPONSIBILITY. To implement this mode use the non-extended initialization routines, such as InitISAM, or make sure that USERPRF_NDATA is 'OR'ed into the user profile mask, userprof, argument of the extended initialization routines.

Note: This mode does not require, or use, the data object definition array (DODA).

c-tree Server Features and Enhancements Dead Client Detection


5.2 Dead Client Detection

A customer reported client-side issues after long client inactivity. After FairCom investigated the situation, it was concluded this was limited to clients connected to a c-tree Server running on certain operating systems with FairCom’s dead client detection feature enabled.

The client problem occurs due to some platforms mishandling TCP/IP OOB (out of band) messages, which c-tree Server uses to determine if the client has been disconnected.

By definition, OOB data is separate from the normal TCP/IP data stream. However, when a c-tree Server was hosted on a platform without proper OOB data support (specifically Windows 98), some of the OOB data was getting into the normal data stream. This insertion of extra bytes of data into the communication stream caused the client application to receive incorrectly formed messages from the c-tree Server, which led to errors and crashes on the client side. Again, this was due to an operating system deficiency and FairCom has consequently implemented the following work-around.

FairCom has introduced a new method of detecting dead clients, using the standard TCP/IP data stream rather than OOB data. This new dead client detection method requires clients compiled with new dead client detection communication capabilities in the current line of c-tree Plus code.

Implementation On the Server side:

• Use the new c-tree Server with improved dead client detection.

• Include the COMPATIBILITY TCPIP_CHECK_DEAD_CLIENTS keyword in the Server configuration.

• FairCom added a new keyword to control the timeout interval: DEAD_CLIENT_INTERVAL nsec, where nsec is the number of seconds of client idle time after which the Server checks the connection for that client. The default nsec value is 1800 (30 minutes), with a minimum of 120 (2 minutes).

NOTE: The timeout interval only controls how often the c-tree Server sends a message to test the connection. Different operating systems use different timeout values on TCP/IP messages, so the actual delay before a dead client is dropped will depend on when the operating system notifies the c-tree Server that the message failed.

On the client side:

• Recompile the new c-tree Plus client code to build a client library with #define ctFeatNEW_DEAD_CLIENT active (default).

• Relink the client application with the new library.

c-tree Server Features and Enhancements Dead Client Detection


Operational Detail This capability is invisible to a properly implemented client application and server. The following paragraphs provide some detail about what was changed.

At connect time, the client indicates to the server whether or not it supports the new dead client detection method. An old client does not indicate this support to the server, and a new client indicates that it supports this feature only if #define ctFeatNEW_DEAD_CLIENT is enabled when the c-tree client library is built. Currently this #define is enabled by default in the c-tree Plus headers.

If dead client detection is enabled (using the COMPATIBILITY

TCPIP_CHECK_DEAD_CLIENTS keyword), the server checks whether or not the client supports the new dead client detection method (as indicated by the client).

• If the client does not support the new method, the server writes a message to CTSTATUS.FCS (for the first occurrence only) indicating that dead client detection is only available for new clients and disables dead client detection for that client.

• If the client does support the new dead client detection method, the client and server use the new dead client detection method for that client.

Additional Technical Detail Both old and new dead client detection methods use a timed select() operation followed by a send() to the client. The send() operation is required so that the subsequent select() can determine whether or not the client has been disconnected.

• The old dead client detection method sent one byte of OOB data from server to client. In normal circumstances this OOB data is invisible to the client (except for the problematic Windows 98 case described above).

• The new dead client detection method sends four bytes in the standard data stream, having a value that can be distinguished from standard data, currently "\xFF\xFE\xFD\xFC". The client must therefore be prepared to discard occurrences of this data when reading its standard data stream, which is why this new dead client detection method is only available for clients that are compiled with #define ctFeatNEW_DEAD_CLIENT.

c-tree Server Features and Enhancements Portability Enhancements


5.3 Portability Enhancements

The following platform-specific enhancements improve the portability of the c-tree Server.

Windows XP 64-bit Port Completed

FairCom is pleased to announce the availability of the c-tree Server for Windows® XP 64-Bit Edition. This latest port from FairCom delivers the extreme performance that is necessary for some of our customers handling large quantities of data, particularly in the scientific and engineering fields.

Microsoft’s new 64-bit client operating system was designed to meet the demands of specialized, technical workstation users who require large amounts of memory and floating point performance in areas such as movie special effects, 3D animation, engineering and scientific applications.

With FairCom customers needing real-time access to gigabytes and even terabytes of data, the c-tree Server running on Windows XP 64-Bit Edition and an Intel® Itanium™ processor delivers unmatched performance.

AIX v5.1 Server Port Completed

In our continuing effort to provide our customers support for the most current and efficient systems, the c-tree Server is now available for AIX 5.1 from IBM.

Apple MAC Server Unique File Detection

The Unique File Detection capability added in c-tree Plus V6.10 has been ported to the c-tree Server for Mac OS 7-9.

MacTCP Dead Client The updates made to the TCP/IP Dead Client Detection capability (described earlier in this document) allowed this capability to be ported to the Mac OS 7-9 platform.

Bound Server Shared Library on Linux

The ability to create a Linux Shared Library version of the Bound Server model was added in this release. To make the Shared Library option available, execute mtmake with ‘s’ as a parameter:

mtmake s

c-tree Server Features and Enhancements Diagnostics and Debug Enhancements


.NET Compiler Support – Microsoft C/C++ Compiler v13.00

We have added support for the most current Microsoft Compiler (.NET – MS C/C++ v13.00). A new option on the m-tree make system creates makefiles with the proper setting in order to take advantage of this support.

As with previous versions of the Microsoft development tools, it is still mandatory to execute a Microsoft-supplied batch file before running mtmake.exe to set the proper environment variables. The file is VSVARS32.BAT, typically located in C:\Program Files\Microsoft Visual Studio .NET\Vc7\bin.

UNICODE ICU 2.1 Libraries Updated – Mac OS X Supported

FairCom introduced support for Unicode data types, key segments, and file names in V7.11. Because of the complexity of the Unicode collation algorithm, and because of the incredible breadth of language and country support envisaged by Unicode, FairCom has chosen to implement Unicode key segments on Microsoft Windows NT/2000/XP using the International Components for Unicode (ICU) open-source development project.

In our latest release, we’ve updated these libraries to ICU v2.10 in order to provide improved Unicode support and have extended Unicode support to Mac OS X.

5.4 Diagnostics and Debug Enhancements

Locking Activity Enhanced

The lock monitor and ctLOKDMP calls already provide information about locking activity. This enhancement adds a total lock request count (across all data and index files including header lock requests and other system generated lock requests such as index node locks), a total count of the number of lock requests that blocked (i.e., went to sleep waiting for a lock), and a total count of the number of blocking lock requests that were denied because of a dead-lock. These are output in CTSTATUS.FCS when the server shuts down.

Optionally at compile time, if ctFeatSrvLock is defined, then the total number of header lock requests, number of blocked header requests (which are both a subset of the total requests and total blocks, respectively), and the total number of locks denied are also reported. Also the lock counts are maintained for each logon session and reported by user at logoff in CTSTATUS.FCS, identified by user ID and node name. These have been made optional since there is some small overhead associated with these additional counters.

Because the #define ctFeatSrvLock must be activated, a special gen of the server is required. Please contact FairCom if this feature is of interest to you.

c-tree Server Features and Enhancements Other Server Enhancements


New CTIDMP Utility The ctidmp utility lists the contents of a dynamic dump file or a specific extent of a dump broken into multiple files with the !EXT_SIZE keyword.

Compile and link this utility with a c-tree Plus Standalone Single-user TRANPROC library.

ctidmp [dump file name]

ctflvrfy – New Index Verify Utility The new c-tree Plus Index Verify utility, ctflvrfy, takes an index name as the parameter and calls the ctVERIFY() and chkidx() functions to allow the user to verify and index, and optionally inspect it at a low-level.

ctflvrfy filename.idx

Use sysiocod to Monitor Disk I/O Activity A call to SetOperationState(OPS_DISK_IO,OPS_STATE_ON) uses sysiocod to indicate if the previous function call to the Server generated any read and/or write activity for data and/or index files. sysiocod is only set if the previous call did not itself cause a sysiocod return value.

A sysiocod between –1015 and –1000 indicates that the feature is on, and no other sysiocod value occurred. There are sixteen possible return values, defined as:

Symbolic Constant

Value Description

ctDISKIO_IDXR 1 Index read from disk. ctDISKIO_DATR 2 Data read from disk. ctDISKIO_IDXW 4 Index write to disk. ctDISKIO_DATW 8 Data write to disk.

A return of –1000 means no data or index disk activity, though there still may have been activity in transaction logs and other Server files. A return of –1015 means there were read and writes on both data and index files.

This statement prints a message as shown:

if (sysiocod <= -ctDISKIObase && (((-sysiocod) – ctDISKIObase) & ctDISKIO_DATW))

printf(“One or more data file writes to disk\n”); else printf(“No data file write to disk\n”);

5.5 Other Server Enhancements FairCom has added the following features and enhancements:



• Improved the Suspend Server option to allow the Server to start in Suspended mode and to include internal server threads.

• Windows Service shutdown now uses Registry setting for shutdown delay.

• Enhanced Bound Server directory support and printf()/exit() customization.

• PAGE_SIZE maximum increased from 32 KB to 64 KB.

• Read-only file mode enhanced to allow file sharing.

• Enhanced performance when encryption is in use.

• FUNCTION_MONITOR logs c-tree Plus error values.

• Added SIGNAL_READY and SIGNAL_DOWN support on Windows platforms plus new user-defined SIGNAL_READY and SIGNAL_DOWN functions.

• Server idle cache-flushing keywords added.

• Block superfile member create/delete during dynamic dump.

• Enhanced automatic recovery – skip questionable indices.

Suspend Server Option Enhanced

A call to Security(………,SEC_BLOCK_KILL) causes logged on clients and dynamic dump threads to be terminated, and keeps users (except for ADMIN) from logging into the system.

Now the internal threads that might open files are also suspended:

• The delete node thread and SYSLOG threads are suspended.

• The variable length space management thread is suspended.

• The checkpoint thread is not suspended since it only searches for open files, it does not open them.

This is to help developers who do not want to terminate the server, but do want to be sure the server is not acting on any files. This enhancement allows the Server Administrator to perform a full backup of files under c-tree Server control without shutting down the server.

New Server Keyword: STARTUP_BLOCK_LOGONS

The new STARTUP_BLOCK_LOGONS Server keyword prevents non-ADMIN user logons when the server is started. Only users in the ADMIN group are allowed to logon.

This feature allows the server to start for ADMIN purposes before authorizing access by non-ADMIN users. This could include creating or adjusting files, adjusting security options, or any other operations that require a functioning Server but are more conveniently accomplished when users are not connected to the Server.

Once the ADMIN work is finished, call Security with the following command to allow all user logons:

SECURITY((COUNT)-1,(pVOID)0,(VRLEN)0,SEC_BLOCK_OFF);



Windows Service Shutdown Uses Registry Shutdown Delay By default when the system is shutting down, Windows allows 20 seconds for a service to stop before terminating the service. When present, the Windows WaitToKillServiceTimeout registry setting overrides the default delay. The c-tree Server Service now checks and uses this registry setting when it is available.

Bound Server Directory Support The c-tree Plus Bound Server Model now supports the equivalent of the c-tree Server LOCAL_DIRECTORY, SERVER_DIRECTORY, and MIRROR_DIRECTORY keywords. The ctINIT structure, passed in the cfg parameter to ctThrdInit includes three new members:

pTEXT locdirc; /* LOCAL_DIRECTORY*/ pTEXT srvdirc; /* SERVER_DIRECTORY*/ pTEXT mirdirc; /* MIRROR_DIRECTORY*/

This change is backward compatible since the ctINIT structure has room for new members.

locdirc and srvdirc are mutually exclusive. If both are initialized, then the local directory overrides the server directory. See the keyword descriptions in chapter 6 of the c-tree Server Administrator’s Guide.

It is not necessary to set these members to non-NULL pointers unless you want to affect where (and how) file names are modified. For example, to prepend “data\” to each non-absolute file name without changing the file name entered in the transaction logs, see the following pseudo-code:

ctINIT cfg; memset(&cfg,0,sizeof(cfg)); cfg.locdirc = “data\\”; ctThrdInit(12,0,&cfg);

c-tree Server SDK and Bound Server: printf() and exit() Overrides Overriding the printf() and exit() functions within the c-tree Server SDK, including the Bound Server Model, has been made easier. Developer can modify two functions, ctrt_Unix_printf() and ctrt_Unix_exit() found in ctsunx_u.c, to control console output and exit routines as shown in the following code:

ctCONV void ctDECLV ctrt_Unix_printf( char *fmt, ... ) { va_list ap; va_start(ap, fmt); vprintf(fmt,ap); va_end(ap); } ctCONV void ctDECL ctrt_Unix_exit( int ret ) { exit(ret); }



The source code for these functions is in ctssup.c for standard c-tree Servers and ctasup.c for client-side applications.

PAGE_SIZE Maximum Increased To 64K The PAGE_SIZE maximum capacity value has been increased to 64K. Prior to this adjustment, the maximum PAGE_SIZE was 32K (32768 bytes). Now, it is possible to set the PAGE_SIZE value as high as 64K (65536 bytes). Both 32K and 64K are large page sizes. Most c-tree Plus developers use page sizes between 2K and 8K, usually based on the page size for the host operating system.

If the PAGE_SIZE specified in the configuration is above 64K, c-tree Server uses the 64K value and writes a message to CTSTATUS.FCS.

The page size determines:

• How much data is read or written by the data file caching system each time data is fetched or put to disk.

• The size of an index file node: The larger the node, the shallower the tree, which generally leads to better performance, but with compressed keys a larger node may mean more time decompressing.

The disk I/O subsystem may respond better to some page sizes than others, so it is best to test different page sizes in your production environment to maximize performance.

This option ties in with the new MPAGE_CACHE keyword to allow long variable-length data records to remain in cache.

Read-only File Mode Improved – Best of Both Ways c-tree’s interpretation of “opening a file in read-only” mode (ctREADFIL) needs some clarification. When a file is opened with ctREADFIL mode, c-tree enforces the fact that “all-users” (who have this file opened) must be in “read-only” mode, and will not allow the file to be opened if anyone else has it opened for writes or updates. In addition, once you have the file opened with ctREADFIL, no one else can open this file unless they use ctREADFIL. This gives you a way to ensure that no data changes will occur to the file while you have it opened in ctREADFIL mode.

This definition sometimes causes confusion for users who what to open the file “read-only” for themselves, say a query program, yet want to allow other programs to continue updating the file. In order to accommodate this situation, support has been added for (ctREADFIL | ctSHARED). Opening a file with both the ctREADFIL mode or’d with the ctSHARED mode will protect the opening application from writes (i.e., enforce read-only), while allowing other programs alternate access privileges.

Therefore c-tree Plus now supports both modes:

• ctREADFIL: restrict all user access to “read-only” ensuring that no one else can modify the file from underneath you.



• ctREADFIL | ctSHARED: enforce read-only for your specific application while allowing others full access.

An attempt to update data or resources to a file opened in ctREADFIL mode will result in a SWRT_ERR(458).

NOTE: This new improvement only applies to client/server and bound server applications. For standalone models, if both ctREADFIL and ctSHARED are turned on, then the ctSHARED is ignored.

File Encryption Performance Enhanced FairCom made changes to our internal encryption logic to prevent extra processing that should improve c-tree Server performance when using basic encryption.

FUNCTION_MONITOR – Log all c-tree Error Numbers The c-tree Server FUNCTION_MONITOR provides a means to route function call information to a disk file. By placing a file name (say, MY_OUTPUT.LOG) after the FUNCTION_MONITOR keyword, all information will be written to this file:

FUNCTION_MONITOR MY_OUTPUT.LOG

When output is routed to a file, an additional feature is gained. All c-tree error codes will also be logged to this file. The screen (console) output does not include these error numbers. This feature is very handy for diagnostics, especially if you are not the primary author of the c-tree client application.

Launch Executables at Server Launch and Shutdown The c-tree Server has the ability to launch (or spawn) an executable by providing an executable name in the server configuration with the SIGNAL_READY and/or SIGNAL_DOWN keywords.

The SIGNAL_READY keyword launches the executable at server startup time, when the c-tree Server is ready.

The SIGNAL_DOWN keyword launches the executable when the c-tree Server shuts down.

Prior to this release, this support was only available on Unix versions of the c-tree Server. We have now added this feature to the Windows (Win32) c-tree Servers.

User-defined SIGNAL_READY and SIGNAL_DOWN Functions Developers using the c-tree Server SDK may now define functionality to occur at Server startup and shutdown through two new Server keywords:

USER_SIGNAL_READY my_parameter USER_SIGNAL_DOWN my_parameter

Each of these keywords passes my_parameter to a user-defined function in ctclbk.c, which the developer should alter to suit the needs of the custom server:



NINT ctusersig_ready(TEXT mystring) NINT ctusersig_down(TEXT mystring)

Each keyword can be specified multiple times with different parameters to accomplish different tasks.

Server Idle Flush Cache Keywords Added The c-tree Server now supports flushing of data and index caches during c-tree Server idle time. c-tree Server launches two idle thread processes at start-up: One thread flushes transaction-file buffers and the other flushes non-transaction-file buffers.

The threads wake-up periodically and check if the c-tree Server is idle to begin flushing. Subsequent c-tree Server activity terminates the flushes. Low priority background threads, such as the delete node thread, do not affect the c-tree Server idle state, but c-tree Plus clients and transaction checkpoints modify the idle status. Note: SQL clients (at this point in time) do not cause the idle status to be modified. This will be rectified in the next update.

The following server configuration keywords modify the characteristics of these idle flush threads:

IDLE_TRANFLUSH <idle check-interval (secs) tran files> IDLE_NONTRANFLUSH <idle check-interval (secs) nontran files> Setting the interval to zero or a negative value disables the thread. The defaults for these intervals are now each set at 15 seconds.

Block Superfile Member Create/Delete During Dynamic Dump c-tree Server contains support for blocking the adding and deleting of superfile members (agents) during the course of a dynamic dump. This feature uses two server configuration keywords to selectively block superfile member adds and deletes.

COMPATIBILITY BLOCK_DDSFM_CREATE – Blocks superfile member creation during a dynamic dump. Attempts to create a superfile member return DDCR_ERR(740) with this keyword activated.

COMPATIBILITY BLOCK_DDSFM_DELETE – Blocks superfile member deletion during a dynamic dump. Attempts to remove a superfile member return DDDR_ERR(741) with this keyword activated.

IMPORTANT NOTE: An application may create or delete superfile members in a superfile host waiting to be dumped while the overall dump is going on. Once the dynamic dump begins dumping the superfile host, blocked operation cannot occur until the end of the entire dump, not just the end of the dump of the superfile host itself. Therefore, the last superfile host listed in the dump script file list will have the shortest blocking period.



Automatic Recovery – Skip Questionable Index Files The c-tree Server automatic recovery routine ensures that all data and index files controlled under transaction processing fully recover when c-tree Server re-starts. There are obscure cases (perhaps a file corrupted outside of c-tree Server control) where the automatic recovery detects a questionable index and cannot fully ensure the integrity of this index. Automatic recovery reports these indexes.

Automatic recovery now skips over these questionable index files and continues processing. Prior to this change, the automatic recovery would stop at the first questionable index and terminate the c-tree Server, preventing recovery.

Now, the recovery continues and properly recovers all files. Questionable index files, detected during recovery, are reported at the end of the process, and the user is instructed to rebuild these indexes.


6. c-tree Plus API Enhancements

FairCom has added new functionality to c-tree Plus, including:

• Added option to check the status of locking on a given record offset.

• Allow EXCLUSIVE file caching in the Standalone Multi-user (FPUTFGET/Peer-to-Peer) operational model.

• Added optional ‘Append to end’ capability – Disable deleted space re-use.

• Added option to rename files under transaction control.

• Added a new mode allowing DoBatch to retrieve only the fixed-length portion of targeted variable-length records.

• Allow non-transaction indices to be dropped.

• Windows DLL: c-tree exported functions now have fixed Ordinal Values

• A new GetCtFileInfo mode returns the first byte offset for data records.

• New GetXtdCreateBlock function retrieves extended file creation block.

• A new LockCtData mode frees ISAM-level locks.

• PutIFileXtd8 was adjusted to allow operations on open files.

• New StopUserAsync function allows the client to continue without waiting for the Server to complete the user stop operation.

• New CtreeAsynchronous function added to determine the status of an asynchronous thread or to cancel the thread.

• New CtreeFlushFileXtd function added mode and parm values allow applications to flush cache for specific files and groups of files.

• Various m-tree and other enhancements.

6.1 Check Lock Status With ctLockRecord

A new LockCtData mode value, ctCHKLOK, checks the lock status of a particular record position for a data file.

If LockCtData(datno,ctCHKLOK,recbyt) returns NO_ERROR then sysiocod holds the lock status.

If sysiocod and uerr_cod are zero, then no lock is held by the calling user.

If sysiocod is non-zero and uerr_cod is zero, interpret sysiocod as follows:

• The low order byte contains the type of lock: ctENABLE(2) for a write lock, or ctREADREC(4) for a read lock.

c-tree Plus API Enhancements FPUTFGET EXCLUSIVE Files Cached


• The high order byte is comprised of one or more of the following attributes bits (left shifted 8 bits):

Symbolic Constant

Return Value

Description

ctISAMLOCK 1 ISAM lock ctTRANLOCK 4 lock obtained or touched in transaction ctTRANOUT 8 lock obtained outside of transaction

The higher order byte may contain other attribute bits that may be found in ctport.h where ctISAMLOCK is defined.

For an example of the usage of ctCHKLOK, a call of the form LockCtData(datno, ctCHKLOK, recbyt) might return the following values:

Symbolic Constant

Return Value

sysiocod Description

NO_ERROR 0 0x0502 ISAM write lock at recbyt obtained inside transaction

NO_ERROR 0 0x0000 no lock held at recbyt

6.2 FPUTFGET EXCLUSIVE Files Cached

FairCom added the ability to cache files in c-tree’s Multi-user Non-Server (Peer-to-Peer) model (FPUTFGET). FPUTFGET specifically does not cache data. Because it is designed to allow multiple users to manage the data simultaneously, all reads and writes to the database are forced (e.g., all reads are done directly from the disk and write operations flush (sync) all data to disk. The internal acronym FPUTFGET stands for "force put/force get ", or in other words, “force write” and “force-read”.

However, when a file is opened in EXCLUSIVE mode, there is no sharing of the file, and therefore no need to force reads and writes. Cache can be used to gain better performance.

As of this update, when a file is opened in EXCLUSIVE mode in c-tree’s Multi-user Non-Server (Peer-to-Peer) model (FPUTFGET), it will be cached.

NOTE: If this file is not properly closed, its header will be marked with a “dirty” flag, and any subsequent open to this file will fail with a FCRP_ERR(14) error, indicating that the file’s data may not be intact.

In order to take advantage of this feature, use either the InitCTreeXtd or the InitISAMXtd extended initialization call. If a non-extended initialization call is used (INTREE or INTISAM) under FPUTFGET, THERE WILL BE NO DATA CACHE ALLOCATED.

c-tree Plus API Enhancements Special Add Record Feature – “Append Only” mode


As always, a data file that is NOT opened EXCLUSIVE will not be cached and therefore not use any of the data cache pages.

COMPATIBILITY NOTE: Prior to this change, it was possible to open the file exclusively, and if the file was not properly closed, reopen the file again without error. This is no longer the case, as described above. Prior to this change, all data written to this file would have been forced to disk, therefore no open error was necessary when the file was reopened. Now, because the data is cached, c-tree must indicate an error condition if the file is reopened after not being properly closed.

6.3 Special Add Record Feature – “Append Only” mode

A special “append only” feature has been added to c-tree’s add record logic. A customer requested the ability to always assure that c-tree records are ALWAYS appended to the end of the file and NOT to reuse deleted space for new records. Typically, when a new record is being added, c-tree will check for any deleted space in the file, and reuse this space by inserting the new record into this spot. For special audit reasons, a user required all records to be added to the end of the file (appended).

In order to take advantage of this feature, a file must be created with this capability. To indicate this type of file, we allow a file to be created with the ctADD2END mode OR’ed into the x8mode of the XCREblk for the data file. We achieve the always add-to-end-of-file behavior by never re-using deleted space. A record can be deleted, and is marked deleted, but we do not add the record to the available pool of records. This forces all adds to occur at the end of the file. Even during rebuild we do not update the space management information. Only compacting the file will reclaim deleted space. At this time we have not provided a way to turn off this behavior once the file has been created with ctADD2END.

6.4 API Function Enhancements

Rename Files under Transaction Control Support has been added to c-tree to allow renaming of files under transaction control. Now, TRANDEP files must be renamed within active transactions. This permits complete transaction control of file renames just as TRANDEP create and delete behave. The feature applies to ctRENFIL and RENIFILs. RENIFIL / RENIFILX require that all or none of the associated files (i.e., data and index files) be TRANDEP; otherwise a FMOD_ERR(48) is returned.

This means that an existing program that performs renames on TRANDEP files (necessarily outside of a transaction) will now receive a TNON_ERR(71).

Without this feature, a transaction-controlled file (TRANDEP or not) cannot be renamed within an active transaction.

NOTE: renaming involving SEGMENTED and PARTITIONED files is not yet supported.

c-tree Plus API Enhancements API Function Enhancements


DoBatch Option Reads Just Fixed Portion of Variable-length Record

In certain circumstances, one might find it desirable to use DoBatch, the c-tree Plus batch function, to retrieve only the fixed-length portion of a set of variable-length (VLEN) records. As an example, a developer might store basic (frequently read) data in the fixed portion of VLEN records then use the VLEN portion to store other information that is less frequently accessed. In this case, when retrieving records using batches, the developer can use the new BAT_RET_FIX return type to the DoBatch function to improve performance by avoiding the overhead of reading the VLEN portion of the selected records.

The BAT_RET_FIX return type operates in the same manner as BAT_RET_REC except that only the fixed length portion of a variable length data record is returned. If BAT_RET_FIX is used for a fixed length data file the results will be the same as using BAT_RET_REC. When BAT_RET_FIX is called for a variable length file, it returns information in the same manner as BAT_RET_REC called for a variable length file except that only the fixed length portion is returned.

The form of the information returned is the same: record position, (entire) record size, fixed length portion. The size information provides the total record size of the variable length record (even though only the fixed length portion is returned). Nowhere in the output is the fixed length size explicitly returned. In this, it is the same as when BAT_RET_REC is called for a fixed length file. It is assumed the application has knowledge of the fixed length size, say through a call to GetCtFileInfo(datno,RECLEN).

ctDROPIDX: Permit a non-TRANPROC index file to be dropped

Previously, a call to ctDROPIDX did not permit a non-transaction-processed index to be dropped. Instead, an IITR_ERR(468) was returned if this was attempted. The code was modified so that no requirements on transaction states are checked if the file does not support transaction processing, extending this feature to non-transaction-processed.

Windows DLL: c-tree exported functions now have fixed Ordinal Values The m-tree make system creates a DEF file for Windows when dynamic load libraries (DLLs) are defined. Prior to this release, the ordinal values defined for each exported function in this DEF file were not fixed. In other words, if a new function was added to the c-tree API, the ordinal values for the exported functions could change, making the new DLL incompatible with existing application executables unless the application was re-linked with the new c-tree library.

Although it is still recommended, and preferable, to always re-link your applications with any new releases of the c-tree library, we have made it easier for users who find this is not always practical. From this release forward, all c-tree functions will be assigned fixed ordinal values in the DLL, ensuring that their reference is reconciled to the same “slot” for all subsequent c-tree releases. This way it is possible to drop in a new c-tree DLL without re-linking the application. All existing c-tree functions will be reconciled at the same location, and any new exported c-tree API calls will be placed at the end of the ordinal list (vector table).



New GetCtFileInfo (GETFIL) mode: Return starting offset data records On occasion it is necessary to calculate the beginning offset within a c-tree Plus data file where the first data record begins. Users may desire to do so for low-level debugging or data verification. In order to streamline the process of computing this value, we have added a new mode, BEGBYT, to GetCtFileInfo.

GetCtFileInfo(datno,BEGBYT) returns the first possible location for a data record in a data file unless the data file is a member of a superfile in which case it returns the position of the last record added to the file.

If this mode is called for an index, the function returns –1 and uerr_cod is set to FMOD_ERR(48).

The position returned is not necessarily an active data record: it may be a resource or a deleted record.

Retrieving the Extended File Creation Block A new function, GetXtdCreateBlock, retrieves the extended file creation block XCREblk. It is declared as follows:

NINT GetXtdCreateBlock(COUNT filno,pXCREblk pxcreblk)

GetXtdCreateBlock fills the structure pointed to by pxcreblk with the contents of an XCREblk corresponding to the extended create attributes for filno. Use GetXtdCreateBlock to retrieve the XCREblk for an existing file so that a new file can be created with the same extended create attributes.

If pxcreblk is NULL, then PNUL_ERR(540) is returned. If filno is not opened, FACS_ERR(26) is returned. If filno corresponds to an index member, KMEM_ERR(23) is returned. Otherwise, the structure is filled in and NO_ERROR is returned.

NOTE: It is legitimate to call GetXtdCreateBlock for files created under V6 or V7 in Standard or Extended format. The resulting XCREblk from a call for a Standard format file can be used in a call to an extended create routine, and the result should be a Standard format (V6 compatible) file.

Frees ISAM Level Locks with LockCtData (LOKREC) LockCtData (short name: LOKREC) frees all ISAM level locks associated with a specific data file (datno) with the lokmod parameter set to ctFREE_ISAM. While this capability was added to allow additional functionality for higher-level c-tree interfaces, this extended capability may be useful to developers at the ISAM level in a multi-user environment.

Allow PutIFileXtd8 Call On Open File Prior to this modification, it was necessary for a file to be closed before you could update its IFIL information stored in the file’s header. Now, PutIFileXtd8 supports the ability to update this header information for an open file.



New Function StopUserAsync added – Stop User Asynchronous An asynchronous stop user function StopUserAsync has been added to c-tree. It is the same as StopUser except that it returns immediately when called by a client without waiting for the server to complete the STPUSR processing.

A client call to original StopUser waits until the server has completed all processing related to that user (i.e., flushing files, closing files). StopUserAsync was added to allow the client application to submit the stop and move on without waiting for the server to complete the stop. As the stop operation is typically the last call made by applications, waiting for the server to complete is unnecessary.

StopUserAsync accepts a mode parameter to convey special user shutdown options. The only mode implemented affects flush operations: if a file is on the NO_SHUTDOWN_FLUSH list (see NO_SHUTDOWN_FLUSH keyword), and if StopUserAsync is called with ctNOFLUSH or’ed into the mode parameter, then the file buffers will not be flushed if the StopUserAsync causes the file closure, and the file will be marked corrupt.

CtreeAsynchronous – New API call added A new API call, CtreeAsynchronous, declared as LONG CTASYNC(LONG

handle, NINT mode, VRLEN bufsiz, pTEXT bufptr), deals with asynchronous threads (including flush threads) launched by an application. CtreeAsynchronous can be used to determine the status of an asynchronous thread or to cancel the thread.

Added the ability to see and cancel asynchronous threads with the CTADMN list user option (just as dynamic dump threads are treated).

CtreeFlushFileXtd Added – New Cache Flushing API call A new API call, CtreeFlushFileXtd, declared as LONG CTFLUSHX(COUNT

filno, NINT mode, LONG parm), has been added to c-tree Plus. Various mode and parm values allow applications to flush cache for specific files and groups of files. See the function description in Section 6.7, “New Function Descriptions”, for more details.

c-tree Plus API Enhancements m-tree Make System Enhancements


6.5 m-tree Make System Enhancements

Some adjustments have been made to enhance the flexibility of the command-line make utility, mtmake.

m-tree’s “custom” Directory Relocated The m-tree “custom” directory has been moved from ctree/custom to custom.xxx. to parallel the bin.xxx, obj.xxx, and lib.xxx directories (where xxx is the directory extension for the operational model selected). As an example, when building the Client-side version of c-tree Plus, the directories created now include: bin.cli, obj.cli, lib.cli, and custom.cli. This enhancement makes it easier to build client, standalone, and similar libraries without overwriting the contents of the custom folder.

Add Unique Extensions to mtmake Directory Names The mtmake.exe make utility allows developers to specify the extension used during the build. This feature is especially useful when building multiple libraries of the same type, such as two or more client libraries.

The parameter ext:xxx, where xxx may be any text string, defines the working directory extension. For example:

mtmake.exe ext:today

creates the following directories:

bin.today lib.today obj.today custom.today

Add Timestamp to mtmake Directory Names Adding the –f switch when running the mtmake utility adds a timestamp on the end of created directory names. This is a convenient way to mark specific builds.

c-tree Plus API Enhancements Other Features and Enhancements


6.6 Other Features and Enhancements

CTSBLDM Rebuilds Superfile Index Members Ordinarily, the procedure to rebuild a superfile is to use ctsbld to prepare the entire superfile, making the data good and returning space used by the indices. Then, each set of ISAM files is rebuilt. This approach is used since a superfile entwines all of the member files and no assumptions are made about the integrity of the members.

A new utility, ctsbldm, now permits the indices associated with a non-corrupt superfile member data file to be rebuilt without having to process the entire superfile. It assumes that the data file contains an IFIL describing all the indices, that the data file is not corrupt, and that the superfile is in good shape except for the indices in question. One possible use for this utility is when recovery cannot repair an index. This rarely occurs, but may be caused by a loop in the index leaf nodes. If this index is a member of a superfile, we can now attempt to rebuild just the indices, and not process the entire superfile. The utility forces all indices associated with the data file to be rebuilt.

The utility operates by opening the data and index files, deleting the index files, and then rebuilding the index files.

ctsbldm <DataFileName> [<Optional PageSize>]

New Index Segment Mode: Schema-based Serial Segments Index segment modes are used to define specifics related to individual key segments that make up an index. Schema segments (e.g., SCHSEG) instruct c-tree to calculate the offsets related to the index segment, as opposed to the developer hard-coding the offset. A new segment mode has been added which allows Schema-based definition of a serial number segment: SCHSRL.

A segment with a mode of SCHSRL is interpreted in a manner similar to SCHSEG: the segment offset is a zero-based relative field number. For example, an ISEG structure with values {3,6,SCHSRL} is interpreted as a serial number segment whose record offset is determined by the location of the 4th entry in the DODA, and whose segment length is 6. Serial number segments can be from 4 to 8 bytes in length. To exceed 4 bytes, the file must use an extended header (i.e., not a V6 compatible file, although the file does not need to be HUGE).

Index Segment Enhanced - Support Segment Offsets > 32767 You now have the ability to define an Index Segment at a location in your record with an offset greater than 32K.

The ISEG structure is comprised of three COUNT fields that contain the segment offset, length and mode. However, a fixed length file or the fixed length portion of a variable length file may exceed 32767 bytes. This modification supports segment offsets exceeding 32767 bytes (but less than 65535 bytes). The structure field type was not changed, but the processing logic deals with segment offsets that “go

c-tree Plus API Enhancements Other Features and Enhancements


negative.” A minus one for a segment offset (equivalent to 65535 for a UCOUNT) still signifies the “end of segments” so that segment offsets can now range from 0 to 65534.

If a value greater than 32767 is stored in a COUNT field, it appears as a negative value; but its absolute value correctly corresponds to the original value. For example, storing 65534 into a COUNT results in a value of –2 when expressed as a COUNT, but –2 cast to a UCOUNT corresponds to 65534.

Prevent False Flags In Fixed Length Data Records c-tree has always used the first byte in fixed-length records as a special indicator for deleted records. Experienced users protect themselves by either reserving space (i.e., a delete flag) or by ensuring a string-type field is the first field in the record (because a valid string field will never have an FF or FE value in the first byte). New users tend to miss this detail, and place integers as the first field in their record. Valid integers may have either FF or FE and therefore cause problems within c-tree.

In order to help protect against this error, c-tree Plus has been enhanced to check the beginning of records for certain values. AddRecord and ReWriteRecord now return FBEG_ERR(553) if a record begins with either a delete flag (0xFF) or a resource mark (0xFEFE).

This capability can be disabled:

• For non-server applications, disable the check by compiling with NO_ctBEHAV_CHECKFIX defined.

• For client/server applications, the configuration keyword COMPATIBILITY NO_CHECKFIX turns off this check.

ctmark Enhanced The ctmark.c sample program permits a new action, aP, which causes a loop of NXTREC – DELREC calls. This was developed for internal use to ensure that index nodes are depopulated quickly for delete node thread testing, but may be of some interest to advanced developers.

c-tree Plus API Enhancements New Function Descriptions


6.7 New Function Descriptions

The following functions were added with this release:

• ctFILELIST

• CtreeAsynchronous

• CtreeFlushFileXtd

• GetXtdCreateBlock

• PartitionAdmin

• SetDataFilter

• StopUserAsync



ctFILELIST Manage file-specific cache features.

SHORT NAME ctFILELIST

TYPE Utility Function

DECLARATION NINT ctFILELIST(pTEXT filnam, pLONG pvalue, pLONG pmember, NINT state, NINT action)

DESCRIPTION ctFILELIST manages the advanced file-specific cache capabilties for standalone and bound server applications. ctFILELIST must be called prior to the creation or opening of filnam for the cache capabilities to take effect. If multi-byte file names are supported (e.g., Unicode), the configuration information (or direct call to ctFILELIST) must have the file names in UTF8 form (i.e., a NULL terminated string).

The state parameter determines which cache capability is invoked, with pvalue and pmember providing supplemental information, as shown in the table below (which assumes the action ctADDfilelist:

state Description ctSPLCACHElist Assign pvalue bytes of dedicated cache to filnam. ctPRICACHElist Loads pvalue bytes from the beginning filnam into cache. ctNO_CACHElist Prevents filnam from using cache. ctPRIINDEXlist Loads pvalue bytes from the beginning of index member

pmember in index file filnam into cache. A negative value for pmember indicates the entire index.

ctNO_FLUSHlist Sets filnam to not flush cache at file close.

The action parameter can be used to add a file to a list, check or change the values associated with a file on a list, or to remove a file from a list. Changes made to an open file have no effect until the next time the file is opened.

action Description ctADDfilelist Add a file to the list.

ctSRCfilelist Return current values for file.

ctCHGfilelist Change values for listed file.

ctREMfilelist Remove file from list.



RETURN

Value Symbolic Constant

Explanation

0 NO_ERROR Successful function

101 INOT_ERR filnam is not a member of the state list.

See Appendix A of the c-tree Plus Programmer’s Reference Guide for a complete listing of valid c-tree Plus error values.

EXAMPLE pTEXT filnam = “custmast.dat”; LONG bytes = 4000000; NINT rcN = 0; /* Set the file to use 4000000 bytes dedicated cache. */ if (rcN = ctFILELIST(filnam, &bytes, NULL, ctSPLCACHElist, ctADDfilelist)) errorReturn(rcN, “Error on SPLCACHE assignment”); /* Set the file to load 4000000 bytes to cache at open. */ if (rcN = ctFILELIST(filnam, &bytes, NULL, ctPRICACHElist, ctADDfilelist)) errorReturn(rcN, “Error on SPLCACHE assignment”)); /* Open the file and load 4000000 bytes to dedicated cache */ if (OpenFileWithResource(-1, &filnam, ctSHARED)) errorReturn(isam_err, “Error opening file.”);

LIMITATIONS The ctFILELIST processing, used to store and retrieve cache parameters, ignores the mirrored portion of a file name. Only the primary file is cached.

SEE ALSO Section 3.4 File-specific Cache Enhancements



CtreeAsynchronous Deals with asynchronous threads launched by an application.

SHORT NAME CTASYNC

TYPE Utility

DECLARATION LONG CtreeAsynchronous(LONG handle, NINT mode, VRLEN bufsiz, pTEXT bufptr)

DESCRIPTION CtreeAsynchronous deals with asynchronous threads (including flush threads) launched by an application. CtreeAsynchronous can be used to determine the status of an asynchronous thread or to cancel the thread.

• handle is the handle value returned by a function that launches an asynchronous thread (e.g., CTFLUSHX).

• mode is ctASYNC_STATUS or ctASYNC_CANCEL.

The bufsiz and bufptr parameters are ignored now. They are intended for future use if it is desirable to return information stored by an asynchronous thread.

At this point, only the thread that launched the asynchronous thread can call CtreeAsynchronous.

RETURN Symbolic Constant Value Description ctASYNC_completed 0 Thread completed successfully ctASYNC_cancelled 1 Thread cancelled ctASYNC_notfound 2 handle not found ctASYNC_badmode 3 mode is bad ctASYNC_running 4 Thread is running

negative value negative Negative of c-tree error code that terminated the thread




CtreeFlushFileXtd Flush cache for specific files and groups of files.

SHORT NAME CTFLUSHX

TYPE Utility

DECLARATION LONG CtreeFlushFileXtd(COUNT filno, NINT mode, LONG parm)

DESCRIPTION CtreeFlushFileXtd extends CtreeFlushFile with mode and parm values to allow applications to flush cache for specific files and groups of files. Calling CtreeFlushFileXtd with mode and parm set to zero is identical to calling CtreeFlushFile(filno).

The following mode options may be appropriately mixed and matched:

ctFLUSHX_ASYNC Launch flush and return immediately ctFLUSHX_ISAM Treat filno as an ISAM data file and flush data and

associated indices (only when filno!= -1) ctFLUSHX_TRAN Only flush TRNLOG and PREIMG files (filno == -1) ctFLUSHX_NONTRAN Only flush non-transaction files (filno == -1) ctFLUSHX_SYSALL Flush all files opened by system (filno == -1) ctFLUSHX_USRALL Flush all files opened by calling user (filno == -1) ctFLUSHX_BYTES Limit flush to specified number of bytes (parm holds the

limit on the number of bytes to flush) ctFLUSHX_PERCENT Limit flush to specified percent of bytes (parm holds a

percentage between 1 and 100) ctFLUSHX_LAST This option only applies to asynchronous flushes that do

not attempt to flush all system files (ctFLUSHX_SYSALL which is the default when filno is passed in as a –1). When used, the asynchronous flush will only flush the file if the caller is the last (i.e., only) client to have the file open. This permits the client to terminate without concern that a file close will take an inordinate amount of time due to the close flushing all the file’s buffers.

Not all combinations of these modes make sense. CtreeFlushFileXtd eliminates conflicting modes before attempting the flush. For example, supplying a valid filno and then specifying ctFLUSHX_TRAN, ctFLUSHX_NONTRAN, ctFLUSHX_SYSALL, and/or ctFLUSHX_USRALL is inconsistent. In this case the filno is used and the conflicting mode option is ignored.

RETURN CtreeFlushFileXtd returns the handle, a positive LONG value, associated with a successfully launched asynchronous thread (if mode contains ctFLUSHX_ASYNC), NO_ERROR if CtreeFlushFileXtd completes successfully (no thread launched), or a



negative error code in the case of a problem.

LIMITATIONS NOTE: The only option available with stand-alone applications is ctFLUSHX_ISAM.



GetXtdCreateBlock Retrieves the extended file creation block, XCREblk.

SHORT NAME GETXCREBLK


DECLARATION NINT GetXtdCreateBlock(COUNT filno, pXCREblk pxcreblk)

DESCRIPTION GetXtdCreateBlock fills the structure pointed to by pxcreblk with the contents of an XCREblk corresponding to the extended create attributes for filno. Use GetXtdCreateBlock to retrieve the XCREblk for an existing file so that a new file can be created with the same extended create attributes.

NOTE: It is legitimate to call GetXtdCreateBlock for files created under V6 or V7 in Standard or Extended format. The resulting XCREblk from a call for a Standard format file can be used in a call to an extended create routine, and the result should be a Standard format (V6 compatible) file.

RETURN If pxcreblk is NULL, then PNUL_ERR(540) is returned. If filno is not opened, FACS_ERR(26) is returned. If filno corresponds to an index member, KMEM_ERR(23) is returned. Otherwise, the structure is filled in and NO_ERROR is returned.

LIMITATIONS Must be called for each physical file.



PartitionAdmin Partition file administration function.

SHORT NAME PTADMIN


DECLARATION COUNT PartitionAdmin(COUNT datno, pVOID partdesc, LONG prawno, COUNT ptmode)

DESCRIPTION PartitionAdmin manages the partitions for datno, the file number for the host data file. partdesc may be either a pointer to a partition member data file name, a pointer to a key value (already transformed) representative of a partition member, or NULL. prawno may either be a “raw” partition number or its value is ignored.

ptmode controls how to interpret partdesc and prawno as well as what action PartitionAdmin should perform. ptmode is formed by or’ing together the appropriate bit values from this list defined in ctport.h:

Symbolic Constant Value Description ptADMINname 0x0001 partdesc is a file name (ignore prawno) ptADMINkey 0x0002 partdesc is a key value (ignore prawno) ptADMINraw 0x0004 use prawno (ignore partdesc) ptADMINpurge 0x0010 remove partition(s) ptADMINadd 0x0020 add partition(s) ptADMINarchive 0x0040 archive partition(s) ptADMINbase 0x0080 modify lower limit on prawno ptADMINnumber* 0x0100 modify limit on number of partitions ptADMINreuse* 0x0200 reuse purged member raw # ptADMINactivate 0x0400 activate archived member(s) ptADMINstatus 0x0800 return member status in sysiocod ptADMINmulti* 0x1000 purge,archive,etc multiple partitions

* The ptADMINnumber, ptADMINreuse, and ptADMINmulti options have NOT been implemented.

NOTE: All calls to PartitionAdmin must specify exactly one way in which the partition will be identified (ptADMINname or ptADMINkey or ptADMINraw) and exactly one operation (e.g., ptADMINpurge or above). If the ptmode parameter is not correctly formed, PTADMIN will return BMOD_ERR(446).



If the ptADMINstatus option is used, then PartitionAdmin returns either an error code (such as PRNG_ERR – partition number is out of range) or NO_ERROR. If NO_ERROR is returned, then sysiocod contains the member status code from this list:

Symbolic Constant Value Description pmSTATUSnone 0 member does not exist pmSTATUSexst 1 member active pmSTATUSopnd 2 member active and physically opened pmSTATUSarhv 3 member archived pmSTATUSpurg 4 member purged pmSTATUSparc 19 member archive pending pmSTATUSppnd 20 member purge pending

There is a subtlety here concerning a return of pmSTATUSnone in sysiocod, and some possible error codes returned by PTADMIN. For example, a NO_ERROR return from PTADMIN(………,ptADMINstatus) and sysiocod of pmSTATUSnone means that the partition number is in the current valid range, but that no such partition member has been created. A PRNG_ERR return means that the partition number is out of the current valid range (whether or not such a partition was ever created).

A partition that is not in use or a partition that has been opened by the host (i.e., not explicitly by the application), and which is not in use by another user, can be archived or purged.

RETURN


Explanation

0 NO_ERROR Successful function

712 PRNG_ERR Partition number out of range

713 PHST_ERR File is not partition host

714 PMBR_ERR File is not partition member

715 PNOT_ERR Raw partition does not exist

716 PXPR_ERR Bad value for partition key

717 POVR_ERR Could not overload partition number

718 PUSD_ERR Partition member in use

719 PPND_ERR Partition member pending purge

720 PPRG_ERR Partition member purged

721 PARC_ERR Partition member archived

722 PLST_ERR Bad partition host list

723 PTRY_ERR Must retry operation




Explanation

724 PCRP_ERR Bad current ISAM position for host

725 PVRN_ERR Partition resource version error

-726 PPRG_COD Duplicate error caused by purged part

727 PFMD_ERR Bad partition file mode settings

728 PSUP_ERR Attempt to use partitioned file without partition support

729 PUNQ_ERR Purged unique global keys encountered

730 PHGH_ERR Partition number out of range - high

731 PRIK_ERR Illegal operation with partition key


LIMITATIONS A raw partition number must be 1 or greater.

Encryption is not supported for partitioned files.

To support transaction processing, a partitioned file must also support transaction-dependent file creation (TRANDEP).



SetDataFilter Specify a data filter for a given data file.

SHORT NAME SETFLTR

TYPE Data Manipulation

DECLARATION COUNT SetDataFilter(COUNT datno, pTEXT condexpr)

DESCRIPTION SetDataFilter specifies that the data filter condexpr should be used for reads from data file datno until a new filter is set or the file is closed. To “turn off” a filter without closing the file, simply call SetDataFilter with an empty string (“”) for the conditional expression.

ISAM level record requests, including sets and batches, skip over records failing the filter transparently to the user. In client/server, this may substantially reduce network traffic.

A low-level data record read returns FFLT_ERR(739) if the data record exists but does not satisfy the filter.

For partitioned files, call SetDataFilter for the host data file and the filter automatically applies to each partition member file.

The condexpr string is of the same as used for conditional index support: it may be a logical expression involving field names from the record schema or a callback designator (e.g., “@mycallback”).

RETURN


Explanation

0 NO_ERROR Filter applied.

48 FMOD_ERR Must be called for a data file.

596 CMIS_ERR No record schema defined for datno.

597 CINI_ERR Could not parse conditional expression.

598 CVAL_ERR Insufficient data: Condition outside of read data portion.

739 FFLT_ERR Low-level: Data record exists but does not satisfy filter.


SEE ALSO Conditional Expressions



StopUserAsync SHORT NAME STPUSRA

TYPE Server Only

DECLARATION COUNT StopUserAsync(COUNT mode)

DESCRIPTION StopUserAsync is the same as StopUser except that it returns immediately when called by a client without waiting for the server to complete the STPUSR processing.

StopUserAsync accepts a mode parameter to convey special user shutdown options. The only mode implemented affects flush operations: if a file is on the NO_SHUTDOWN_FLUSH list (see NO_SHUTDOWN_FLUSH keyword), and if StopUserAsync is called with ctNOFLUSH OR’ed into mode, the file buffers will not be flushed if the StopUserAsync causes the file closure, and the file will be marked corrupt.

RETURN


Explanation

0 NO_ERROR Successful disconnect.

133 ASKY_ERR User cannot be found (has it been shut down?).

162 SGON_ERR Server communication link has been disturbed.

410 USTP_ERR User is not logged on when StopUserAsync was called.

SEE ALSO StopUser


7. c-tree Driver Enhancements This chapter details new features, enhancements, and fixes for the c-tree® Drivers: c-tree® ODBC Driver, c-tree® Crystal Reports™ Driver, and c-tree® Driver SDK.

7.1 Performance Enhancements

FairCom profiled the ODBC Driver in record retrieval situations and found ways to optimize the database and SQL engine code. FairCom found that the more columns selected in a query, the more overhead is involved to return the records to the ODBC application. FairCom improved the efficiency of the SQL engine's column data retrieval logic.

A performance test involved reading records in physical order from a fixed-length data file containing 55000 512-byte fixed length records with 6 associated indices by executing the following SQL statements:

SELECT * FROM patients

c-tree Plus FPUTFGET 2 seconds

ODBC Driver 8 seconds SELECT FirstName, LastName FROM patients

ODBC Driver 3 seconds

Executing the same SQL statements after applying optimizations produced the following results, a significant enhancement in performance:

SELECT * FROM patients

ODBC Driver 4-5 seconds SELECT FirstName, LastName FROM patients

ODBC Driver 1-2 seconds

Allow overriding number of records reported to SQL engine The c-tree ODBC Driver automatically optimizes a query by examining the columns and criteria in the query and the indexes defined for the tables. In some cases, a developer or end-user might have additional knowledge about the distribution of key values in the index or other properties of the data that can be used to optimize a join. The Driver provides a way to override the order in which it executes a join operation.

When opening a file, the c-tree ODBC Driver now looks in the data dictionary directory for the file <table>.nrc, where <table> is the symbolic table name. If this file exists, the Driver and reads an integer from the file that is used as the number of records in the data file rather than using the actual number of records in the file. By

c-tree Driver Enhancements Microsoft Access-Specific Issues


overriding the actual number of records reported to the SQL engine, it is possible to change the order in which the query is executed, which might be desirable in some cases.

7.2 Microsoft Access-Specific Issues

These issues apply to using the ODBC driver with Microsoft Access.

“Invalid index definition” Due to Duplicate Index Names The c-tree ODBC Driver now checks for the presence of duplicate index names and in this situation generates unique index names. The check is not comprehensive, but simply compares each index name to the first index name. This test is sufficient to avoid what is likely to be the most-common problem encountered, in which all indexes are set to the same name.

#Name? (memory allocation error) Reading With Access 2000 A customer reported #Name? errors when viewing a linked ODBC table using Access 2000. The problem was traced to a memory allocation error returned by the ODBC driver manager when reading the data. The memory allocation error was caused by the following situation:

• Access 2000 called SQLSetStmtAttr() with the SQL_MAX_LENGTH parameter and a value of 2 GB (this indicates that the maximum length of data to return is 2 GB).

• Access 2000 called SQLGetData() with a 201-byte buffer to retrieve a string field of length 202 (201 plus one null-terminator byte). Because the buffer size was not sufficient to hold the entire string value, the SQL engine set *pcbActual to indicate to the application that more data remained to be read. The SQL engine, however, used the SQL_MAX_LENGTH value (probably attempting to enforce SQL_MAX_LENGTH set by the application) rather than the actual remaining length of the data. As a result, the driver manager attempted to allocate a 2 GB buffer, resulting in the memory allocation error. The fix is to adjust the SQL engine so that it returns the minimum of SQL_MAX_LENGTH and the remaining data length.

7.3 Crystal Reports-Specific Issues

These issues apply to using the c-tree ODBC Driver with Crystal Reports.

INNER JOIN Support Added Crystal Reports for Visual Studio .NET generates INNER JOIN syntax. For example, instead of the equijoin syntax expected by the ODBC driver:

SELECT * FROM mytableA, mytableB WHERE myColA = myColB

Crystal Reports for Visual Studio .NET generates this INNER JOIN syntax:

c-tree Driver Enhancements Microsoft Visual C++ 6-Specific Issues


SELECT * FROM mytableA INNER JOIN mytableB ON myColA = myColB

The c-tree ODBC Driver now provides compatibility with Crystal Reports for Visual Studio .NET by supporting INNER JOIN syntax.

7.4 Microsoft Visual C++ 6-Specific Issues

These issues apply to using the c-tree ODBC Driver with Microsoft Visual C++ 6.

Optionally Disable Floating Point Unit Exceptions By default, the c-tree ODBC Driver enables the divide by zero FPU (floating point unit) exception so that it can detect when a divide by zero error occurs. A customer requested the ability to avoid enabling this exception to work around a problem with Microsoft's .Net Font class constructor, which raises an ArithmeticException exception if floating point unit exceptions have been enabled.

The c-tree ODBC Driver now checks the registry for a string value named EnableFPUExceptions under the registry subkey:

HKEY_CURRENT_USER\Software\ODBC\ODBC.INI\FairCom 32bit Driver

If this string value exists and is set to No, the ODBC driver does not enable FPU exceptions.

Microsoft Visual C++ 6 Query Designer fails to read records with "Catastrophic error" error.

When using the Microsoft Visual C++ 6 Query Designer with the FairCom ODBC Driver as the data source, the Query Designer fails to read records and displays a "Driver not capable" error if the following is true:

• The table contains a fixed-length string field with a defined length of over 255 bytes.

• The field value consists entirely of spaces, or consists of one or more spaces and is padded with 0x0 (NUL) bytes.

The ODBC Driver represents a fixed-length string field with a defined length of over 255 bytes as a MEMO field. When reading memo fields, the Query Designer attempts to determine the length of the field by calling SQLGetData() with cbValueMax set to 0. The ODBC driver detects this situation and sets the return value to the full field length. However, the ODBC driver treats a field value for a fixed-length string field consisting of spaces as described in (b) above as a NULL value, so when the Query Designer calls SQLGetData() with a non-zero length to retrieve the data, the ODBC driver returns a length of SQL_NULL_DATA. This discrepancy (initially returning the full field length and then returning SQL_NULL_DATA) leads the Query Designer to display the "Catastrophic error" error.

The ODBC driver now returns a length of 0 when cbValueMax is 0 and the field value is NULL, which prevents this error.

c-tree Driver Enhancements Driver Fixes


Microsoft Visual C++ 6 Query Designer fails to open table with "Driver not capable" error when using the ODBC driver in multi-user non-server mode (ODBC and SDK)

When using the Microsoft Visual Studio Query Designer with the FairCom ODBC Driver as the data source, the Query Designer fails to open tables and displays a "Driver not capable" error if the ODBC Driver is configured to use multi-user non-server mode. The Query Designer successfully opens the table if the ODBC Driver is configured to use client/server mode.

The cause of the error is that Visual Studio calls SQLSetConnectOption() with the SQL_AUTOCOMMIT attribute and the driver returns "Driver not capable" when operating in multi-user non-server mode.

The ODBC Driver in multi-user non-server mode indicates to applications that it does not support transactions for two reasons:

• To avoid an error that occurs when using Access 2000 and multiple DAO recordsets

• Because the underlying c-tree Plus database engine doesn't support transactions in multi-user non-server mode.

To work around this error, the ODBC Driver now checks if it was loaded by Microsoft Visual Studio (by checking if the module file name is "msdev.exe"), and if so the driver indicates that it supports transactions in multi-user non-server mode.

7.5 Driver Fixes

The following fixes apply to the c-tree ODBC Drivers (ODBC) or the c-tree Crystal Reports Driver (CR).

Access Violation when Updating Variable-Length Table (ODBC) A customer reported an access violation when attempting to update a record using Microsoft Access. The situation is unusual because the file in question is a variable-length file containing some records in which some of the variable-length fields do not exist at the end of the record. The ODBC driver considers this a valid situation and when reading records treats such nonexistent fields as having NULL values. However, the nonexistent fields led to a problem when updating the record.

The driver was setting the fields in the record buffer but did not increase the used record length in the following situation:

a) The table is a variable-length data file and one or more variable-length fields do not exist at the end of the record. Note: this situation can only occur for records added using c-tree Plus (not by the ODBC driver).

b) The update causes a nonexistent variable-length field to be set to a single delimiter byte.

c) The updated field fits in the record buffer (based on the total buffer size) but exceeds the used record buffer size.

c-tree Driver Enhancements Driver Fixes


Because the driver did not increase the used record length, when the driver attempted to set the value of a field following a field such as described above, a negative length was used in a copy operation, leading to an access violation.

The solution is to increase the used record length to account for the newly-set field.

Field Not Found for Key Segment Leads to Access Violation (CR) If a key segment definition specifies an absolute byte field offset that does not correspond to a field defined in the DODA, the Crystal Driver detects this situation (and displays a warning if the driver’s “c-tree Plus Debug” option is enabled) but does not set the field information for that index, resulting in an access violation due to an uninitialized value returned to Crystal Reports. This situation can arise if the alignment being used by the Crystal Driver differs from the alignment on which the absolute byte field offset is based (this discrepancy can result in the Crystal Driver determining that the specified absolute byte offset refers to a gap residing between two fields).

To prevent an access violation when this situation is detected, the Crystal Driver now always displays an error message and returns an error (preventing the table from being opened). The error message that the driver displays in this situation is:

ERROR! Table tablename: No field found for key segment (soff, slen, smod).

Where tablename is the name of the table, soff is the segment offset, slen is the segment length, and smod is the segment mode.

Crystal Reports Access Violation "memory cannot be read" Error (CR) Refreshing a report that involves criteria based on a string field can lead to an access violation with an error message indicating that memory could not be read. The problem was noticed when refreshing a report containing subreports, and was due to reading past the end of a buffer into an unallocated region of memory when the Crystal Driver examined range information passed to it by Crystal Reports.

To resolve this problem, the driver now performs a string compare rather than a memory compare when the range limits refer to a string field.


8. Critical, Serious, and Other Fixes The following issues represent critical issues and other fixes corrected in recent builds.

8.1 Critical Issues

Dynamic Dump extent size overrun A condition causing a Dynamic Dump’s file to exceed the defined “extent” size has been fixed. A customer reported that after setting the Dynamic Dump !EXTENT size to 2GB, a backup file was created around 3GB. Internal compare logic was repaired.

Bound Server Crash due to Thread Issue FairCom resolved an exception violation with the Bound Server model with adjustments to the PTHREAD code. Any developer experiencing instability with the Bound Server model should take note of this fix.

Dynamic Dump DSIZ_ERR(443) FairCom corrected an issue related to the DSIZE_ERR(443) during a dynamic dump. The dynamic dump process incorrectly reevaluated the size of some files, causing a DSIZE_ERR. The process was corrected to properly evaluate the file size. In addition, detailed diagnostics information pertaining to a DSIZE_ERR is now included in CTSTATUS.FCS.

All users who experienced the DSIZE_ERR(443) during a Dynamic Dump should take note of this adjustment.

8.2 Serious Issues

Reported 8770 error resolved An obscure internal issue resulting in an 8770 error has been resolved. An extreme case was reproduced where files were aggressively being opened and closed. This issue involved c-tree’s internal handling of virtual files. Any user who has experienced an 8770 error should take note of this fix.

Dynamic Dump Wildcard Recursion Check A precautionary measure has been taken related to the wildcard support for filenames listed for Dynamic Dumps. A user reported that an enormously large Dynamic Dump file was created, to the extent that they ran out of disk space. In this case, it appeared that the wildcard logic was caught in an infinite loop, backing up the same file over-and-over. Adjustments have been made to the Dynamic Dump wildcard logic to prevent this from happening. If, perhaps due to OS subtleties, the wildcard logic

Critical, Serious, and Other Fixes Other Issues


repeats a file name, the Dynamic Dump will issue a warning to the CTSTATUS.FCS file and continue on to the next file name after the current wildcard.

Resolved Core Dump During Dynamic Dump Recovery FairCom corrected a core dump during the recovery processing at the end of the ctrdmp processing. An internal flag required by ctrdmp, but not required by a normal application, was corrected.

Core Dump with Segmented Files / FPUTFGET / ctLOCK_DIRECT The Standalone Multi-user model (FPUTFGET) contained an error that caused an exception violation when trying to lock a record/header in a segmented file. This issue only applies to FPUTFGET with ctLOCK_DIRECT (the default locking strategy on Unix). This issue has been resolved.

8770 Error - High Word Contamination in Calls to Non-Huge Files FairCom adjusted code that resulted in either an 8770 server error or perhaps a KMAT_ERR(3) on a unique key delete. Low-level API calls with a record position (recbyt) as an input parameter did not force the high word to zero for non-huge files. If a call returning a record position was not followed by a ctGETHGH call, it might leave an invalid high word value for the next call. This was not an issue for huge files, but was specific to non-huge files.

8.3 Other Issues

Unicode fix: Heterogeneous C/S Unicode support A problem related to the use of Unicode in heterogeneous environments has been addressed. Prior to this fix, it was possible for read operations to get a no-hit because of improper handling of the byte ordering (specific to Unicode). This problem has now been resolved. Users who have encountered problems with Unicode in heterogeneous environments should take note of this fix.

Minor Superfile Fixes (0722) Several issues with superfiles under the c-tree Server were corrected. Customers encountering issues with superfiles controlled by the c-tree Server should review these issues.

• Corrected rebuild issue in client/server with SERVER_DIRECTORY set.

• Properly create indexes that did not previously exist.

Dynamic dump recovery BNOD_ERR(69) A problem which caused BNOD_ERR(69) errors during a Dynamic Dump recovery has been addressed. The issue involved a dynamic dump recovery BNOD_ERR(69) with an index file and/or misaligned data records. When dumping a file, it is



sometimes necessary to put the file into the dump stream in more than one large piece. The dump stream may contain one or more additional file extensions (not to be confused with the dump stream itself being broken into extents). The code to append the extensions onto the files being recreated during the dynamic dump recovery processing used an incorrect file mode that forced the extension onto the end of the file being recreated instead of at the specified offset (near the end of the file). The solution was to changed the file mode from “ab” to “r+b” which permitted the seek operations to be effective.

READ_ERR(36) - Dynamic Dump Recovery Superfile Create A problem which caused a READ_ERR(36) during a dynamic dump recovery has been addressed. This issue was qualified to the creation of superfile members during a dynamic dump. A superfile, that has already been included in the dynamic dump stream file, has additional superfile members added (i.e., file creates of superfile members) during the dynamic dump, while other files are still being dumped, could have caused this problem.

Encrypted Superfile Rebuild Core Dump Resolved A qualified situation resulting in a superfile rebuild (CTSBLD) crash has been fixed. The problem was qualified to encrypted superfiles. User who might have experienced problems rebuilding encrypted superfiles should take note of this fix.

Dynamic Dump Wildcard Fix An issue related to using wildcards for a Dynamic Dump has been addressed. When returning to a parent directory while doing the wildcard search, c-tree Server checked the current directory name for two path separators and overwrote the first with 0x0. For example, c:\faircom\dir1\ became c:\faircom. However, this doesn't account for the case where the path doesn't have two separators. For example, faircom\dir1 was left as faircom\dir1. This led to the generation of file names with incorrect paths, causing the search to miss some subdirectories. Now c-tree Server properly handles this case, so faircom\dir1 becomes faircom, and the filenames are correct.

Dynamic Dump of HUGE files over 4GB DEXT_ERR(599) fixed A problem related to backing up huge files that exceeded 4GB has been resolved. As dump stream files were being created, a DEXT_ERR(599) was unnecessarily generated when the file size exceeded 4GB.

CMPIFILX FUSE_ERR(46) and RBLIFILX NSCH_ERR(199) Errors Additional focus was necessary for files created with c-tree’s Transaction Dependent, TRANDEP, mode related to rebuilds and compacts. Logic was adjusted to prevent a number of problems trying to use CompactIFileXtd and RebuildIFileXtd with TRANDEP files. Specifically a FUSE_ERR(46) running CompactIFileXtd and a



NSCH_ERR(199) running RebuildIFileXtd. Any user who might have encountered these errors should note this fix.

r-tree Client issue resolved An r-tree customer found that certain search mechanisms do not read the entire variable length record when calling a c-tree Server. The resolution involved updating the c-tree Server to be compatible with the internal function being called by r-tree. This issue is now resolved.

Custom Server API fix Developers using the c-tree Server SDK may implement their own API calls through the internal CUSTOM functions. FairCom corrected a reported “byte-flipping” problem within these CUSTOM functions.

Alignment Error fixed – ITIM_ERR(160) and SDAT_ERR(445) An internal alignment error that produced either a ITIM_ERR(160) or a SDAT_ERR(445) has been resolved. This issue was reproduced with the c-tree Crystal Reports Driver. The problem was directly related to scheme-based segments.

Dynamic Dump Restore Fix A problem when using a file name in the dynamic dump script that is different from the name used to (first) open the file has been solved. This “first open” name is the name stored in the transaction logs. The file was recreated by the restore using the name in the script, but it was not properly restored to the clean transaction time by the “rollback” inherent in the restore process. If no !REDIRECT options modify the location of the files recreation, and if the name in the script does not match the name in the log, then an automatic redirection to the name used by the log occurs. That is, the file is recreated using the name in the log file.

NOTE: One case still exhibits the symptom: if the name in the script does not match the name in the log, and the redirection is based on the name used in the script, then the redirection does not permit the “rollback” to find the file. The redirection must be based on the file name (and path) used in the log.

NOTE: The status output of the restore process now uses the name of the file actually used when it is recreated as opposed to the name in the script. That is the message:

DR: recreating file: <name of file>

uses the actual file used to recreate the file, and not the name found in the script. Therefore explicit redirection, or the automatic redirection introduced with this bug fix could change this status output from what would have previously been output.

Dynamic Dump RCHK_ERR(66) Advanced Encryption Issue An issue related to the use of Advanced Encryption techniques with Dynamic Dump, which resulted in an error RCHK_ERR(66), has been resolved. When advanced



encryption using a block-encoding scheme is applied to the transaction logs, the dynamic dump log extraction routine did not ensure that the extraction was a multiple of the encoding block size. This caused erroneous decryption during the dynamic dump restore.

Dynamic Dump – cancel caused a problem A memory violation caused by canceling a Dynamic Dump has been fixed.

TRANBEG ctAUTOSAVE issue resolved The Begin transaction function was not properly recognizing the ctAUTOSAVE mode. This has been rectified.

Server - Infinite loop in NEWREC for Superfiles A problem involving superfiles has been fixed. When a NEWREC operation was called for a superfile member, it was possible for the c-tree Server to go into an endless loop. This issue, in some cases, would report a LEOF_ERR(30) in the CTSTATUS.FCS file. This issue has been resolved.

Superfile Create error not properly reported A subtle condition which caused a failed SuperFile create to not report an error has been solved. While testing the FairCom Crystal Reports Driver, we discovered a failed superfile member create did not return a proper ISAM error code from CREIFIL because the “new” space reclamation code for superfile members that are deleted reset the isam_err code and the marker for a changed current record position (autopos). We saved the required values before the space reclamation operation performs its own ISAM operations, and then restore them in order to solve this issue.

FPUTFGET BIDX_ERR(527) A user reported a error (BIDX_ERR(527)) in c-tree multi-user standalone mode (FPUTFGET) during heavy, multi-user index adds. This “No leaf node found” condition was qualified to high speed index adds being conducted by multiple users at the same time. Specifically, by the time an adder starts back up the tree after a node split, it was possible for two or more levels to be added to the index. The code before this fix only handled one additional level. The solution was: if the addition of new tree levels causes a parent child link to become outdated, the path from the newest root to the node is recomputed by re-traversing the tree.

Standalone FPUTFGET Huge File SEQ8_ERR(689) Fixed Under an extremely heavy load, it was possible for c-tree’s Standalone Multi-User model (FPUTFGET) to get a SEQ8_ERR(689)error with Huge files when reading a file’s header. This error was limited to the extended header related to Huge files and now has been resolved.



LOCLIB Syntax Error Resolved A syntax error was reported related to missing multi-byte string conversion routines (UNICODE) when compiling with LOCLIB. This issue has been resolved.

PRDS_ERR(34) During PreviousKey Call To Server A rare condition causing PreviousKey to return an error PRDS_ERR(34) has been resolved. This issue was limited to client/server models and was fixed by simply adjusting a retry counter.

Internally, adding a key could interfere with PreviousKey on the same index. When PreviousKey moves to a predecessor node, it checks that the predecessor’s successor is the current node. Adding a key at just the right moment could cause a node split that in turn previously caused this check to fail. Now the check is retried.

AutoRecovery/PermIIndex L65 and terr(8798)errors addressed Internal issues within the c-tree Server auto recovery logic and the PermIIndex function were addressed to prevent L65 and terr(8798) errors. Internal considerations for TRANDEP files needed further consideration in order to resolve these issues.

Automatic Recovery/TRANDEP Superfile Member Delete c-tree Plus supports the ability to undo the delete of a file created with the TRANDEP and RSTRDEL attributes. While testing recovery after file deletes with TRANDEP and RSTRDEL attributes, FairCom encountered a terr(1964) or terr(1966) during automatic recovery if a member of a superfile is deleted with the TRANDEP attribute (not the RSTRDEL attribute). The terr implies that the file number assigned to the deleted file is being reused, and the recovery code requires that no file number be reused during recovery. To correct this error, when the superfile member is deleted, the file number is marked so that it does not appear available for reuse.

This situation is very rare, since the superfile must be created as a Transaction Dependent (TRANDEP) file, which is not the default, and it is not typical for applications to delete superfile members.

Automatic Recovery/Superfiles BSUP_ERR(411) Addressed During automatic recovery tests in the FairCom lab, an error BSUP_ERR(411) was generated related to superfiles. Internally, the file mapping logic code to recreate a superfile host and its associated directory index required an adjustment to resolve this error.

Set Function Target Buffer Vulnerability Resolved The set creation functions (FirstInSet, etc.) now use a temporary key buffer of the size MAXLEN to prevent key transformation from overflowing the buffer.

c-tree SET functions that required a target parameter were vulnerable if the application passed a target buffer whose length was less than the defined key length



of the associated index (keyno). c-tree could access undefined memory when attempting to transform the target information for the full length of the key.

A side benefit of this change is that the target buffer remains untransformed, allowing other operations that require an untransformed target.

OpenISAMContext: Infinite loop with pending dropped indices In an obscure situation, the OpenISAMContext function could infinitely loop. If a call to DropIndex was issued during an open transaction, a subsequent call to OpenISAMContext could hang. This issue has been repaired.

Because of the newness of DropIndex, and the un-likelihood of calling OpenISAMContext within the same open transaction after a drop, this issue is considered rare.

CTRBLEX – Rebuild Example program link resolved An adjustment was necessary to m-tree’s make system related to the DEF files used to export functions. When compiling the CTRBLEX (rebuild example program), the CTRDEF function was not being exported properly.

Mac OS X 10.1 Compatibility Some adjustments to the Cocoa interface and some shutdown details were required to allow the c-tree Server to shut down properly under Mac OS X V10.1.

EstimateKeySpan and NbrOfKeysInRange V6/V7 The EstimateKeySpan and NbrOfKeysInRange functions had an error when the client and server were compiled with different MAXLEN values (e.g., a V7 server defaults to MAXLEN of 1024 while a V6 client defaults to a MAXLEN of 255). The server misinterpreted where the second key value was found in the communications buffer. The server now correctly interprets the key values.

Flush Variable-length Space Management Buffers CtreeFlushFile, in the Standalone Multi-User model, flushed the buffer pages used internally for variable-length data file space-management only when it was called to flush all files. Now, CtreeFlushFile also flushes the space management information associated with a single file.

End Of Notes

faircom update guide

Documents