Content Manager OnDemand Database Tables

From CMOD.wiki
Jump to: navigation, search

IBM CMOD Internal Database Tables

This article is a reference of the contents of database tables inside Content Manager OnDemand. The data on this page is to be considered informational, and may change from version to version, platform to platform. There are also constraints on some fields and values, meaning that unless you fully understand how the values in specific fields work, you may combine features or options that are NOT compatible with each other, leading to unexpected, erroneous, or unintended behavior, up to and including problems with stability and/or reliability, and/or data loss. Use this information at your own risk. This page aims to be a more detailed and complete version of the IBM CMOD System Control Tables.

arsag - Application Group Table

This table contains all of the configuration data for Application Group Configuration in Content Manager OnDemand.

Field Sample Values Type Description
NAME System Log, AccountsPayable String Name of the Application Group. No special characters allowed.
DESCRIPTION System Log, Accounts Payable String Description of the Application Group.
AGID 5001,5938,6938 Integer Unique Numeric Identifier for Application Group. Used as a relation to ARSAGFLD, ARSAG2FLD, etc.
AGID_NAME SL,HGA,IBA String Three-character unique Alpha identifier for Application Group. Used in database ("Segment") names, as a filespace name in TSM, and in the cache filesystem.
DB_SEG 1,2,3,1000 Integer Used as a counter for the next database segment (table) number.
ANN_TYPE ? ? Determines the Annotation Type for an Application Group.
DB_EXP_DATE 90,365,2557 Integer Number of days to retain the documents loaded into this Application Group for. This field stores the Life of Data and Indexes value from the Administrative Client.
DB_MGRT_DATE 365,720,2557 Integer Duration after which the database segement (aka table) is migrated to secondary storage. Rarely

used.

SM_CACHE_DOC 90,366,720 Integer The period for which data should be cached after being loaded.
RESGRP 1,2,3,40,30000 Integer Used as a counter for the next Resource Group number.
LOAD_ID 1,359,497295 Integer Used as a counter for the next LoadID number.
LOG 1,5,6,38 Integer Vectored Bitfield storing the selected options on the Message Logging tab.
LOAD_ID_SUFFIX FAA, GAA, HAA String Incremented for every 1 million LOAD_ID values. Incremented left-to-right from F to Z, A to Z, then A to Z until exhausted -- allowing for over 13.5 trillion loads in total.
MGRT_CACHE_DATE 30,45,90, 180 Integer Number of days after loading when the documents are migrated to secondary storage.
REIMPORT_EXP_DATE 30,90,180 Integer Number of days after which a re-imported database table is expired.
SID 1,80,88,837 Integer The Storage Set ID Number. A unique integer is defined for each new Storage Set defined in the Administrative Client.
EXPIRE_TYPE x'41', x'4C', x'53' HexChar The Expiration Type for the Application Group:
  • x'41' = "D" = Document
  • x'4C' = "L" = Load
  • x'53' = "S" = Segment
SEG_ROWS 2500000,10000000 Integer Maximum number of rows per database table (aka Segment).
SEG_ID 1,2,3,1000 Integer Used as a counter for the next Segement (aka Database Table) number.
OBJ_SIZE 10000,50000 Integer Maximum Object Size per compressed object loaded into the Application Group.

See ARS0141E for information on tuning Compressed Object Sizes in CMOD.

TYPE 17,529,1553 Integer Vectored Bitfield reflecting the state of the buttons in the Storage Management tab.
UPD_USERID ADMIN String The UserID of the last user to perform an update to the Application Group.
UPD_DATE 06/21/2015 08:15 Date The date of the last update to the Application Group configuration.
LAST_DOC_DATE 15727 Integer The numeric internal date of the oldest document that exists in the Application Group. Deprecated in CMOD v9.0.
MIGR_SRVR_STR
SM_CACHE_DELTA 90,720,3652 Integer The delta between the previous cache retention setting (SM_CACHE_DOC) and the current value. This allows CMOD to properly retain (or expire) data in the cache that was loaded to the cache filesystem prior to the change of the cache retention setting. Prior to CMOD v7.1.2.8, data loaded into the cache before changing an Application Groups cache retention setting would expire on the schedule that was in effect at the time it was loaded. The value in this field is used during cache expiration processing with arsmaint to help determine if data should be removed from the cache or not. See IBM CMOD APAR on SM_CACHE_DELTA - March 2009
DATABASE_NAME
SM_CACHE_RES -1,90,365,2557 Integer Duration to cache Resource Groups in Content Manager OnDemand.
FTI_SERVER ? String String that defines configuration data for Full Text Indexing.
UPD_DT 2015-06-21 08:15:05.385839 Database Date Date of last update using the internal database date format, introduced in CMOD v9.0.
LAST_DOC_DT 2016-04-14 17:38:30.285285 Database Date Date of oldest document that exists in the Application Group. Introduced in CMOD v9.0.
LAST_LOAD_DT 2016-04-15 04:05:48.385921 Database Date Date of the last load into the Application Group in the internal database date format.
LAST_QUERY_DT 2016-04-16 10:46:03.958183 Database Date Date of the last query performed on the Application Group, in the internal database date format.
LAST_RETR_DT 2016-04-15 04:07:19.727563 Database Date Date of the last document retrieval from the Application Group in the internal database date format.

arsag.type - Storage Management Configuration

Database Value Button Description
1 Tablespace Type: None Specifies that the next table for this Application Group should be placed in the default tablespace, which is USERSPACE1 on DB2.
16 Tablespace Type: SMS Specifies that table should be placed in a new SMS table space.
65535 Tablespsce Type: Automatic Specifies that tables should be placed in tablespaces automatically, as determined by your Database Administrator.
524288 Compress tables/Tablespaces If this bit is on, the database will compress table/tablespace data.

arsag2fol - Application Group to Folder Mapping

This table is used to map individual Application Groups to individual folders. An Application Group (AGID) may appear multiple times in multiple Folders (FID). ARSAG2FOL also maps individual fields in a folder to specific fields in Application Groups.

Field Sample Values Type Description
FID 5104, 5319, 5502 Integer Folder ID field. Uniquely identifies a specific folder - see the ARSFOL table.
AGID 5028, 5195, 5425 Integer Application Group ID field. Identifies Application Group(s) that are mapped to the folder identified above.
AID 0 Integer Unknown/Unused
Folder_Field 1, 2, 3, 4 Integer Identifies the field number inside the folder - see the ARSFOLFLD table.
AppGrp_Field1 1,2,3,4 Integer Identifies the number of the field in the Application Group that the folder field is mapped to.
AppGrp_Field2 -1 Integer Unknown/Unused.

arsagfld - Application Group Fields Table

This table contains a list of all of the fields defined in CMOD Application Groups.

Field Sample Values Type Description
AGID 5001, 5357, 3525 Integer Specifies the Application Group that the fields are related to. Relates to ARSAG.AGID.
Field 0,1,2,3,4,5 Integer Sequential number representing the field number, starting at 0.
Name RPT_ID,CUST_NUM String The text name of the field.
Type x'49', x'53', x'62' HexChar The hex values represent single-byte characters which describe a field type.
  • x'49' = "I" = Integer
  • x'53' = "S" = String
  • x'62' = "b" = Database Internal Date
Qual x'42', x'44' HexChar Application Group Field Qualifier - further defines the field type.
  • x'42' = "B"
  • x'44' = "D"
Mask 1, 2, 32770 Integer This is a vectored field that stores the information in the field checkboxes in the Application Group Field Information Tab.
String_Type x'00', x'46', x'56' HexChar The hex values represent single-byte characters which describe a string type.
  • x'00' = None (Field is not a string type.)
  • x'46' = "F" = Fixed
  • x'56' = "V" = Variable
String_Len 0, 1, 2, 5, 10, 60 Integer The length of the string field. '0' if the field is a date or numeric field like Integer or Float.
String_Case x'4D', x'55' HexChar Represents a string field's case.
  • x'4D' = "M" = Mixed
  • x'55' = "U" = UPPER
  • x'4C' = "L" = lower
Leading $, #, <space>, - String A list of characters to remove from the FRONT of the string before inserting them into the database.
Strip $, #, <space>, - String A list of characters to remove from ANYWHERE in the string before inserting them into the database.
Trailing $, #, <space, -> String A list of characters to remove from the END of the string before inserting them into the database.
Map_Field NULL, -1, 0, 1, 25 NULL or Integer

arsagfld.mask

CAUTION: Changing these values without knowledge of how they inter-operate or conflict with each other is dangerous and not supported.

Checking the associated checkbox increases the value by the specified value below:

Value Checkbox Name Description
1 Indexed The Field Type is set to 'Indexed' when enabled, Filter when disabled.
2 Updatable The value for this field can be updated by users or administrators with sufficient privileges.
4 Segment Specifies that this field should be used to segment the database tables, for faster access. All Application Groups should have a Segment Field for maximum performance.
8 Application ID Field This field is used to identify different Applications Defined in the Application Group.
16 Expire Date This field is used to calculate the expiration date of the document when the Expiration Type is set to 'Document'.
32 Log Inserts the value of this field into the System Log for DocGets (retrievals) - message number 66
64 User Exit Sends the field name and value to a user exit.
128 Unknown Unknown
256 Partition Partitions the database based on this field. Only one field can be selected as the partition field.
512 Cluster Builds a cluster index if the database engine supports it.
1024 Reference Used by the OnDemand Distibution Facility to determine if the document should be distributed.
2048 Unknown Unknown
4096 Page Count This field contains the number of pages in the document.
8192 Document Size This field contains the size of the document in bytes.
16384 Unknown Unknown
32768 Unknown Unknown
65536 Lockdown This field contains the number of holds that exist for a document when CMOD Enhanced Retention Management is enabled.
131072 CFS-CMOD Required for interoperation with FileNet if your server is enabled with CFS:OD.
262144 Full Text Index Specifies that this field is indexed by the full-text indexing option for Content Manager OnDemand.

arsann - OnDemand Document Annotation Table

Value Sample Value Type Description
ID ? Integer Annotation ID Number
USERID ADMIN, USER String The User ID that created the annotation.
AGID 5229, 5918 Integer The unique Application Group Identifer that the Annotation belongs to.
DOC_NAME 123FAAA String The name of the object the annotated document is stored in.
DOC_OFF 0,83852 Integer The offset from the beginning of the object, in bytes, that the annotated document begins at.
DOC_LEN 38582,1894 Integer The length of the document, in bytes, for the annotated object.
COMP_OFF 478295, 915753 Integer The offset from the beginning of the compressed object, in bytes, that the annotated document begins at.
COMP_LEN 35638,2682 Integer The length of the compressed document, in bytes, for the annotated object.
TIME_STAMP Deprecated Integer In versions prior to CMOD v9.0, the timestamp when the annotation was created.
PAGE 1,2,50,80000 Integer The page number that the Annotation should appear upon.
TYPE ? Integer Describes the type of annotation.
X_OFF ? Integer The X offset of the annotation.
Y_OFF ? Integer The Y offset of the annotation.
RESERVED ? VarChar Reserved for future use.
TEXT_BUF ? LongVarChar The text of the annotation, up to 32 kilobytes in size.
IMAGE_BUF ? Binary Object An image annotation up to 1 megabyte in size.
TABLE_NAME ABC1, DEF2 String The name of the table that the annotated document is located in.
DOC_EXP_DATE Deprecated Integer In versions prior to CMOD v9.0, the date when the document (and by extension, the annotation) was due to expire.
DOC_EXP_DT 2019-05-15 00:00:00.000000 Integer In versions after CMOD v9.0, the date when the document (and by extension, the annotation) was due to expire, in the database's native timestamp format.
TIME_STAMP_DT 2019-05-15 00:00:00.000000 Integer In versions after CMOD v9.0, the date when the annotation was created, stored in the database's native timestamp format.

arsapp - OnDemand Application Definition Table

Value Sample Value Type Description
NAME System Log, CServPDF VarChar Name of the Application
DESCRIPTION Customer Service PDFs VarChar Description of the Application
AGID 5158 Integer Numeric Application Group ID that the Application belongs to
AID 5821 Integer Numeric Application ID
DOC_TYPE x'42', x'55' Character Type of document to store in this Application (see table arsapp.doc_type below)
DOC_COMP_TYPE x'44',x'46' Character Compression method for the document data
RES_COMP_TYPE x'4E',x'4F' Character Compression method for the resource data
IDX_TYPE x'41',x'47' Character The type of indexer to use for this data
COMP_OBJ_SIZE 100,9999 Integer Size of the compressed objects that CMOD builds at load time
PID Unknown Integer Unknown
ALIAS Unknown VarChar Unknown
FIXED_VIEW N/A Character Large Object (32k) Contains information about how to view line data
INDEXER N/A Character Large Object (32k) Stores the indexing parameters for indexing documents with ACIF, ARSPDOCI, or Xenos
PREPROCESSOR N/A Character Large Object (32k) Stores pre-processing commands for manipulating metadata before loading
UPD_USERID ADMIN VarChar The User ID of the user that last updated the Application
UPD_DATE 119287486874 BigInteger The UNIX-timestamp of the time the Application was last updated.
DB_EXP_DATE 16494 Integer It is currently unknown how this date value is used.
APPL_TYPE 0 Integer It is currently unknown how this integer value is used.

arsapp.doc_type - Document type to store

Database Value ASCII Character Document Type Description
x'41' A AFP IBM's Advanced Function Presentation - print output from mainframes
x'42' B BMP Windows Bitmap Format
x'47' G GIF Compuserve Graphics Interchange Format
x'4A' J JPEG Joint Photographic Experts Group
x'4C' L TXT Line data - also used for System Log data. ASCII or EBCDIC text.
x'4D' M IMG Xerox Metacode aka 'DJDE'
x'4E' N None There is no document data, just metadata. Primarily used for the System Log.
x'50' P PCX PC Paintbrush image format
x'52' R PDF Adobe's Portable Document Format
x'54' T TIFF Tagged Image File Format
x'55' U User-Defined A type of data for which OnDemand has no native indexer or viewer. (Word, Excel, Zip, MP3, MP4)

arsapp.*_comp_type - Data compression type

Database Value ASCII Character Document Type Description
x'44' D Disable Don't compress data during storage or transport
x'46' F OD77Lite Faster version of OD77 (but compresses less well)
x'4C' L LZW12 Lempel-Ziv-Welch compression - use OD77 or OD77Lite instead (deprecated)
x'4E' N None Don't compress data at load time, compress during retrieval
x'4F' O OD77 Content Manager OnDemand's purpose-built compressor for AFP & Line Data
x'5A' L LZW16 Lempel-Ziv-Welch compression - use OD77 or OD77Lite instead (deprecated)

Why would you want to disable compression? Some data types stored in OnDemand are already compressed -- JPEG, PDF, GIF, or even user defined data types like MP3 audio or Zip files. Attempting to compress these files usually results in no real savings in storage space (and in some cases, may even make the file bigger) and waste considerable CPU time.

What's the difference between 'None' and 'Disable'? The 'none' option doesn't compress the file when it's loaded into CMOD, but WILL attempt to compress it before sending it to the end user at the time it's retrieved. There so few examples of why this would be beneficial, that I can't even think of any to tell you.

arsapp.idx_type - Indexer for the Application

Index_Type Mappings
Indexer ASCII Letter Table Value
ACIF "A" 65
Generic "G" 71
Internal "I" 73
None "N" 78
OS390 "3" 51
OS400 "4" 52
PDF "P" 80
Unknown "U" 0
XML "L" 76

arsres - IBM CMOD Document Resource Table

One of the ways that Content Manager OnDemand acheives best-in-class compression is to perform de-duplication on loaded and indexed documents. These duplicate portions of documents are bundled into 'resources' which are stored separately from the data itself, referred to at the time a document is retrieved.

IBM CMOD Resource Table Structure
Column Name Type Description
RID INTEGER Unique Resource ID
AGID INTEGER Numeric Application Group ID that the resource belongs to - See: Application Group Table
PRI_NID INTEGER "Primary" Node ID where the resource is stored
SEC_NID INTEGER Deprecated
COMP_TYPE CHARACTER Compression Type - See: Data Compression Types
ADD_DATE BIGINT Date the resource was added, as the number of seconds since the UNIX Epoch, midnight January 1st, 1970 UTC
COMP_SIZE INTEGER Compressed Size in bytes
DECOMP_SIZE INTEGER Decompressed Size in bytes
EXP_DATE BIGINT Expiration date of the resource, as the number of seconds since the UNIX Epoch, midnight January 1st, 1970 UTC
HOLD SMALLINT Is the resource part of a document being held by IBM CMOD Enhanced Retention Management?
ADD_DT TIMESTAMP Date & Time the resource was added in database-native time format.
EXP_DT TIMESTAMP Date & Time the resource is expected to expire, in database-native time format. This date will be updated as new loads are added to Content Manager OnDemand.
HASH_VAL CHARACTER Cryptographic Hash of the contents of the resource bundle.
CRYPT_STATUS SMALLINT Encryption status

arsseg - IBM CMOD Segment Table

The segment table keeps track of how Application Group tables are segmented -- generally by the 'Maximum Segement Size' parameter in the Application Group configuration, in the advanced pane. Dates in this table are formatted as per the Application Group date format that is selected as the Segment Field (by the checkbox 'Segment' in the Field Information Tab in the Application Group configuration in the IBM CMOD Admin client. This means that old-style date/time formats use the UNIX epoch format (number of seconds since January 1st, 1970, UTC) and the old-style CMOD date format (number of days since January 1st, 1970). In versions of IBM CMOD v9.0 and higher, field names that end in _DT use the internal database timestamp format.

IBM CMOD Segment Table Structure
Column Name Type Description
AGID INTEGER Application Group ID number
TABLE_NAME VARCHAR Table name in AGID_NAME+DBSEG format from ARSAG table
START_DATE BIGINT The minimum value of the segment field (date) in this segment / table.
STOP_DATE BIGINT The maximum value of the segment field (date) in this segment / table.
POST_DATE BIGINT The date the segment/table was created.
CLOSED_DATE BIGINT The date the segment/table was closed, either by becoming full (ARSAG.MAX_ROWS) or being forced closed by the IBM CMOD arstblsp utility.
REIMPORTED_DATE BIGINT The date the table was re-imported back into DB2 if it was migrated out to secondary (TSM) storage.
LAST_UPDATE BIGINT The date of the last update to the database table (insert / delete / update to rows).
LAST_BACKUP BIGINT The date of the last backup in the segment field format.
LAST_STATS BIGINT The last time database statistics were run by the arsdb & arsmaint utilities.
MASK INTEGER A vectored field containing configuration data about the table.
INS_ROWS BIGINT The number of rows inserted into this table.
UPD_ROWS BIGINT The number of rows updated in this table, by the arsdoc update command.
DEL_ROWS BIGINT The number of rows deleted from this table, either by expiration processing, or the arsdoc delete format.
MOD_ROWS BIGINT The number of rows modified in this table.
MAX_ROWS BIGINT The maximum number of rows that this table has ever contained, this value is provided by ARSAG.MAX_ROWS at creation time.
DATABASE_NAME VARCHAR On partitioned database systems, the name of the database this table is stored.
FTI_SERVER VARCHAR The Full Text Indexing server that contains the full text index for documents contained in this table.
START_DT TIMESTAMP The minimum (oldest) date of documents stored in this folder, in database-native timestamp format.
STOP_DT TIMESTAMP The maximum (newest) date of documents stored in this folder, in database-native timestamp format.
POST_DT TIMESTAMP The date this table was created, in database-native timestamp format.
CLOSED_DT TIMESTAMP The date this table was closed (by being full, or by the IBM CMOD arstblsp command), in database-native timestamp format.
REIMPORTED_DT TIMESTAMP The date the table was re-imported into the database (after being exported to secondary storage), in database-native timestamp format.
LAST_UPDATE_DT TIMESTAMP The date the table was last updated, in database-native timestamp format.
LAST_BACKUP_DT TIMESTAMP The date the table was last backed up, in database-native timestamp format.
LAST_STATS_DT TIMESTAMP The date the table last had database statistics collected, in database-native timestamp format.

SA2 - IBM CMOD System Load Table

The IBM Content Manager OnDemand System Load Application Group contains information on loads into the server - and while this is convenient in itself, it's especially useful for running quick reports. The System Load table is named SA#, starting with SA2.

IBM CMOD System Load Table Structure
Column Name Type Length Description
LOAD_TIME Integer 4 Integer representing the UNIX timestamp for the load (unused in IBM CMOD V9.0+), see LOAD_TIME_DT below.
APPLGRP_NAME Varchar 60 Name of the Application Group the data was loaded into.
APPL_NAME Varchar 60 Name of the Application the data was loaded into. (Implies specific indexing and compression settings defined in the Application.)
LOAD_ID Varchar 100 The IBM CMOD Load ID for the load.
TOTAL_ELAPSED Double 8 The elapsed time in seconds and fractions of a second for the entire load to complete.
INDEX_ELAPSED Double 8 The elapsed time in seconds for the indexing portion of the load.
TOTAL_DOCS BigInt 8 The total number of documents inserted into the database in this load.
TOTAL_PAGES BigInt 8 The total number of pages in all documents loaded as part of this load. (Where supported.)
DOC_COMP_TYPE SmallInt 2 Document Compression Type - see arsapp.doc_comp_type
DOC_INPUT_SIZE BigInt 8 Input size in bytes of the file to be loaded.
DOC_OUTPUT_SIZE BigInt 8 Size of the data stored in CMOD after compression.
RES_TYPE SmallInt 2 Describes the resource type.
RES_RID Integer 4 The Resource Identifier for the loaded document. May be duplicated if the resources match a previous load.
RES_COMP_TYPE SmallInt 2 Describes the resource compression type - see arsapp.doc_res_type
RES_INPUT_SIZE BigInt 8 Resource size before compression.
RES_OUTPUT_SIZE BigInt 8 Resource size stored after IBM CMOD compression.
LOAD_AGID Integer 4 The numeric Application Group ID of the loaded file. Included because Application Group names are allowed to be changed.
LOAD_AID Integer 4 The numeric Application ID used to load the file. Included because Application names can be changed.
LOAD_NAME Varchar 11 The 'Doc_Name' portion of the Load ID.
LOAD_PRI_NID Integer 4 The Primary Node ID of the Load ID.
LOAD_SEC_NID Integer 4 The Secondary Node ID of the Load ID. (Deprecated and unused.)
LOAD_START BigInt 8 The start date and time of the load in UNIX timestamp format.
LOAD_STOP BigInt 8 The end date and time of the load in UNIX timestamp format.
DOC_BGN_DATE Integer 4 The earliest document date in the Load.
DOC_END_DATE Integer 4 The latest document date in the Load.
INDEX_TYPE SmallInt 2 Describes the indexer that was used to index the data in the load file - see arsapp.idx_type
APPL_ID Character 1 Unknown.
LOCKDOWN SmallInt 2 Determines if the load was automatically held if Enhanced Retention Management is enabled.
CFSOD SmallInt 2 Describes if CFS:OD was used on the Load.
INPUT_FILENAME Varchar 1024 The name of the file that was loaded.
DOC_NAME Varchar 11 Internal object name of the log file.
DOC_OFF Integer 4 Internal offset inside the object name of the log file.
DOC_LEN Integer 4 Internal length of the log file inside the object.
COMP_OFF Integer 4 Internal compressed offset of the log file.
COMP_LEN Integer 4 Internal compressed length of the log file.
ANNOT Character 1 Was there an annotation added to the log file?
COMP_TYPE Character 1 Compression type of the log file.
RESOURCE Integer 4 Resource ID of the log file.
PRI_NID Integer 4 Primary Node ID where the log file object is stored.
SEC_NID Integer 4 Secondary Node ID - deprecated and unused.
LOAD_TIME_DT TimeStamp 10 A timestamp of when the file was loaded in the database's internal date/time format.
LOAD_START_DT TimeStamp 10 A timestamp of when the file started loading into CMOD, in the database's internal date/time format.
LOAD_STOP_DT TimeStamp 10 A timestamp of when the file finished loading into CMOD, in the database's internal date/time format.
DOC_BGN_DT Date 4 A timestamp representing the earliest document date in the load, in the database's internal date format.
DOC_END_DT Date 4 A timestamp representing the latest document date in the load, in the database's internal date format.
INPUT_FILE_SIZE BigInt 8 Size of the loaded file in bytes.