• External index and query (EIQ) Products are based on index and query processing layers maintained external to data sources, where the indexes use cleansed, transformed and standardized data read from data sources and then discarded -- data remains stored in data sources

• EIQ Product layers reside behind or in front of corporate firewalls, can be locally, regionally or centrally located, and absorb the heavy load of maintaining indexes, processing queries, monitoring changed data , etc.

• At least twelve change data capture (CDC) options are used to read source data to maintain indexes, ranging from real-time to batch

• Indexes typically have the same structures or schema as the respective data sources; relational, non-relational or unstructured - EIQ Product indexes do not normally need structure or schema transformation

• EIQ Products are accessed through standard drivers (ODBC, JDBC, OLE-DB, etc.) and  Web Services, and queries submitted in standard query language (SQL)

• Query processing yields a list of pointers back to raw data in data sources -- record numbers, indexed key(s), URLs, RDFs or file positions, etc.

• Raw data is retrieved from data sources through standard or proprietary user-level access, and cleansed, transformed and standardized for eventual presentation to calling applications

  OTHER APPROACHES

Data warehouse, conventional federated adapter and enterprise search (to a lesser extent) systems are all conventional approaches to the same challenge of dealing with data; where it resides; how to access it; how to cope with "dirty data", typos, etc.; different types; different standards; different formats; different security; different locations; different owners; different systems; etc.

Each conventional approach has its advantages and disadvantages. WhamTech combined technologies from these three approaches to provide dataless hybrid products, retaining the advantages and overcoming the disadvantages of each approach.

For structured database-type queries, query success is the same as, if not better than data warehouses and considerably better than conventional federated adapters.

For unstructured text and metadata search, query success is similar to advanced search engines.

The reason for the high query success is that EIQ Products uniquely have indexes that comprise cleansed, transformed and standardize representations of data that is read from data source logs, or some other means, and then discarded.  This enables queries submitted to the indexes to be highly successful. Further, EIQ Products have advanced indexing including text search, fuzzy match, aggregation, calculation, compound, join, denormalized, embedded value and link indexes.

EIQ Products retain the advantages of data warehouse systems:

• Clean data -- indexes and results data

• Multiple indexes and types

• Query processing: multiple options

• Security: data and access

…and at the same time, retain the primary advantage of conventional federated adapters:

• Data remains at source

  LEGACY DEVELOPMENT

EIQ Products derive their unique index and query processing technologies from previously marketed database and search products.

WhamTech and its predecessors developed a relational very large database (VLDB) technology, called D the Data Language and later, Thunderbolt, that was extremely fast compared to other similarly configured database systems.  This appealed to a niche data processing intensive market; however, the real technology differentiators were the unique index and query processing technologies now used in EIQ Products, which index and process queries against almost any and all data sources.

From an operational and structural point of view, the relational index and query management system (RIQMS) embedded in EIQ Products, is a conventional relational database technology, with virtual tables, indexes, and typical database operation commands and queries.  It is NOT a memory-resident system; NOT a read-only system; NOT a fully inverted database system; and NOT a retrieval-based storage system.  There are, however, significant performance and capability differences that distance EIQ Products' RIQMS from other database and related technologies:

Unique Method of Query Execution

EIQ Products RIQMS has a unique method of isolating, connecting, arranging (sorting), processing (updating), and presenting (displaying) data.  This unique technology enables real-time data isolation and access; no matter the database size, number of concurrent users, or query complexity. Unique Combination of Technologies EIQ Products RIQMS speed and advanced capabilities arise from a unique combination of three binary-level methodologies involving the three "Bs" of computing that are normally associated with static data warehousing, not on live data sources, as EIQ Products provide for, and would be very difficult to improve on:

• Balanced binary trees that scale to billions of records, terabytes of data

• Virtual bitmap representations of intermediate and final query result-sets - these can be integer lists or actual bitmaps, depending on field data node-level "data density"

• Boolean operations on bitmaps

Balanced binary trees are a technology from the 1960s and the attraction then, as it is now, is that binary searches are considered to be the fastest method of searching ordered lists[1], however, there are a number of practical problems associated with balanced binary trees; all of which WhamTech has solved (and that is one of the secrets to our success):

• PROBLEM: Levels tend to get very deep, whereby a binary tree consisting of a billion nodes, for example, needs 30 levels; this translates into time to traverse
  WHAMTECH SOLUTION: WhamTech's implementation does not conform to the conventional n = log2(x+1) balanced binary tree rule, where n = number of levels and x = number of nodes.  Instead, WhamTech's implementation "leaps" levels to make binary tree traverse-time measurable in microseconds rather than milliseconds or seconds

• PROBLEM: Rebalancing and rotation after an insert or delete can take considerable time and a very large number of nodes can be affected
  WHAMTECH SOLUTION: The maximum number of nodes that need rotated is just over 100; however, typically only 10 to 15 nodes are rotated after an insert or delete

• PROBLEM: A worst-case scenario of deletion of a top node, which is faced by almost all tree structures WHAMTECH SOLUTION: This is one of the easiest operations for EIQ Products RIQMS to deal with, as it fits the technology well With EIQ Products RIQMS, when a balanced binary tree is subjected to a query, the result is an integer list set of pointers to raw data in the data source.  For subsequent query-related operations, this set is either used directly or is converted to a Collection; an example is shown in the following diagram T1:

T1: Index binary tree with associated virtual bitmap (integer list) and physical bitmap