Using Engines For Performance
Performance issues are often structural, and cannot be solved by optimizing specific queries or calls. Actually, they are often caused by doing several things during a request, that can just as well be done in an asynchronous way before or after the request. Moreover, the asynchronous implementation of these operations often avoids doing these operations many times all over again. It could be considered as some kind of caching at the level of the information system.
In a structured database scheme, the detailed information describing a specific entity is often spread out over many database tables. Aggregating this detailed information for a set of instances requires a time-consuming join over many tables to be performed for every instance of the set. This is fundamentally time-consuming, and cannot just be solved by optimizing the query.
This joining of detailed information can for instance be done by a separate asynchronous engine aggregating all detailed information for every instance, and putting this in a single detailed information table with a single entry for every instance of the data entity. In this way, the aggregation is not performed during the user request, which may also serve as a caching mechanism, avoiding that the aggregation is repeated many times.
Of course, every time (part of) the detailed information is modified, the asynchronous engine should notice this, and update the detailed information table. This seems cumbersome for information which is changed at a rapid pace. However, the original issue typically arises for a gradual buildup of a massive amount of information, that needs to respond swiftly to queries of client systems.
When a piece of detailed information is submitted to an information system, the storage of this information should often be distributed over many entries in various database tables. Moreover, based on this new information, it is often necessary to update the status of existing entries in database tables, even tables which do not participate in the storage of this detailed information.
This insertion and update of detailed information cam for instance be done by a separate asynchronous engine entering all the submitted information in the appropriate structure of the database scheme, and updating all the related statuses. In this way, the structured representation and status update is not performed during the user submission, which may also serve as a caching mechanism, avoiding that this status update is repeated many times.
In order to minimize the computation effort during the actual request, the information may be stored as flat as possible, for instance a single string field storing the entire message as a blob. Due to this dedicated simple structure, the preliminary storage could be performed in a separate table, where the entries would be transferred by the engine in an asynchronous way to the proper structure. This would also entail another performance advantage that the dedicated submission table would always remain small.