EIQ ADAPTERS™ in Detail

External Index and Query (EIQ) adapters™ are federated data access adapters that  are unique in that they externally INDEX and execute queries against these external indexes for all indexable data and information - structured database queries and unstructured text searches - WITHOUT submitting queries to the data source, WITHOUT copying and storing data elsewhere and WITHOUT changing data schemas.

EIQ Adapters™ are NOT conventional federated adapters, NOT data warehouses, NOT enterprise search only, and NOT Extract, Transform and Load (ETL) tools.

EIQ Adapters™ are NOT conventional technology.

WhamTech and its predecessors developed unique high performance index and query processing technologies over the last 25 years as part of their own relational database management system (RDBMS), called Thunderbolt.  As the RDBMS was so fast, WhamTech used it to develop unstructured text search technologies for its search engine products for the Web, enterprise and specialized search, which by definition, operate externally for files, documents, including Web pages, and e-mail.  With EIQ Products™, WhamTech brought the power of its advanced index and structured query processing technology and unstructured text search to databases and data sources other than its own.  For the range of EIQ Products™ offered by WhamTech, please visit products.

EIQ Adapters™ enable radically improved/changed processes (not just better technology)

WhamTech’s EIQ Adapters™ are not conventional, as they take a very different approach to major problems facing almost all large organizations: How to integrate, share and interoperate data and information in near real-time without (a) creating a huge additional super-infrastructure, e.g., massive data warehousing, (b) overloading existing systems, e.g., conventional adapters in federated systems, and (c) losing the ability to execute structured database queries, e.g., enterprise search?

EIQ Adapters™ technology offers the advantages of the above three approaches without the disadvantages associated with each of them. EIQ Adapters™ not only enable data and information (hereinafter collectively referred to as data) virtualization, federation, integration and interoperability within and between organizations, but can also “turbo charge” existing data source access and add significant features such as:

  • Complex query processing
  • Heuristic data mining
  • Pre-aggregated and pre-calculated indexes for direct queries
  • Monitoring key performance indicators (KPIs) and performance
  • Querying across and between databases, files, documents and e-mail
  • Link mapping and link analysis (connecting seemingly unconnected data and information)

EIQ Adapters™ include a highly flexible middleware/sub-middleware product, called EIQ Federation Server™, that resides between applications/middleware and other EIQ Adapters™. EIQ Federation Server™ can also access other EIQ Federation Server™ to allow indefinite performance, scalability, load balancing, redundancy and failover.  Business rules can be applied within EIQ Adapters™ to enable the best combination of data to be queried and passed back to applications, providing true organization-wide real-time data virtualization, federation, integration and interoperability. EIQ Adapters™ are accessed by applications, including middleware, through standard ODBC, JDBC, OLE-DB, Web Services and a native driver.  Standard Query Language (SQL) or WhamTech's native language, TQL, can be used to submit queries.  The following diagram EID1 illustrates where EIQ Adapters™ reside in the application/data source architecture:


EID1: EIQ Adapters™ reside between applications/middleware and data sources

There are some similarities to conventional approaches, but the differences become clear as the process is described in more detail:

  • Discover data sources using ODBC calls, if available, local DBA and/or other means
  • Profile data using raw indexes, which consist of trees and lists (see Technologies for more information).  The trees are histograms of domain values and are used as data profiles.  The lists do not need to be retained, however, a possible near-future option may allow these lists to be used to build the initial clean production indexes, without having to re-read the data source.
  • Develop data quality transforms from the data profiles generated by raw indexes, which cleanse, transform and standardize data read for building and maintaining indexes.  The data profiles allow a "before" and "after" iteration process.
  • Read source data using one or more of twelve ways
  • Clean, transform and standardize data used for indexes
  • Build and maintain indexes using the same index schemas as data sources - no major schema transforms
  • Discard data
  • Map standard data model to EIQ Indexes™ - can have more than one standard data model mapped
  • Applications/middleware access EIQ Products™ as though a data base through standard drivers and/or Web services
  • Present a single virtual indexed view of all data sources to applications based on standard data model(s)
  • Execute SQL queries for structured database queries and unstructured text search almost 100% in EIQ Indexes™
  • Generate a list of pointers, URLs, RDFs, etc., to raw results data in data sources
  • Use pointers to retrieve raw results data from data sources through user-level access with appropriate authentication and security
  • Clean, transform and standardize raw results data
  • Consolidate or otherwise post-process results data from multiple data sources
  • Present results to applications/middleware in any format

The above process is illustrated in some detail in the following diagram EID2:


EID2: EIQ adapter process in detail, in this case, an EIQ SuperAdapter

  EIQ adapter technology key points (others are listed under 28-point comparison)
  1. Virtual, private CLOUD-like access through multiple distributed universal and uniform INDEXES and query engines for structured queries and text search on ALL data sources, from mainframes to Web documents to email - seen as one consistent data source conforming to one or more standard data models.
  2. Data warehouse query and results quality, and performance, due to:
  • Cleansed, transformed and standardized indexes (and result sets)
  • Multiple indexes available, including fuzzy, pre-aggregated and pre-calculated, join and Link Indexes
  • Complete control over query processing
  • Combined structured queries and text search
  1. Obtain results when data sources are unavailable through index inversion.
  2. Archive options available through (i) storing changed data in a separate database, (ii) date-time stamping indexes and/or (iii) embedding changed data in indexes.
  3. Leave source data in place in original format and index schemas as per original data source - only INDEXES mapped to standard data models.
  4. Continue to use legacy applications and data, but allow for:
  • Modern application access to legacy data
  • Legacy application access to modern databases
  • Use as a bridge/transition/migration tool from legacy to modern systems
  1. Scale through non-tiered and/or multi-tiered, independently maintained indexes with single-point for an application and/or multi-point access for middleware, including Web services and XML.
  2. High performance queries:
  • Typically, 10 - 100 times faster than other data-based systems
  • Use the three "Bs" of computing - binary index trees, bitmap result-set representations and Boolean arithmetic operations on both physical and virtual bitmaps (see TECHNOLOGIES for more information)
  1. Near real-time updates immediately available to queries and monitoring for alerts/notifications - watch-lists, event-triggers, subscriptions and thresholds.
  2. Create almost any and all indexes, such as pre-aggregated, pre-calculated, composite, fuzzy, advanced text, Link Indexes™ and Embedded Value Indexes™, which allow data and metadata to be stored in indexes.
  3. Using a combination of indexes and the appropriate standard data model terms, create VIRTUAL data warehouses/data marts for multi-dimensional queries.
  4. Link Indexes™ add-on option to enable link mapping, allows for more advanced queries such as complex joins, degrees of separation, physical, logical and ontology model discovery, merge, combination, validation and presentation, and link analysis.  The combination of normal content indexes and Link Indexes™ externally manages obvious/direct links and reveals non-obvious/indirect links among disparate data and information across multiple disparate data sources.
  5. Other add-on options including open source or commercial categorization software, WhamTech's eDiscovery tool, Teracase™, Intelligent Search's fuzzy match, open source GATE entity extraction and WhamTech's advanced text search.
  6. Significant benefits compared to conventional adapters in federated systems, data warehouse or enterprise search - combines the best aspects of these approaches and overcomes the worst

More information on WhamTech products, click here.

 SmartData Fabric™:  unleash the value of data.

For more related information, please visit the pages listed below.