Page History

Understand the datasets engine architecture.

Reference → Modules → Datasets → UI | Engine

Overview

The Dataset Engine orchestrates the internal mechanics of database operations through a multi-layered architecture. This reference covers the deep technical aspects of how the engine processes requests internally.

The engine orchestrates:

• Synchronous and asynchronous operation coordination
• Thread management and request queuing
• Client/server domain resolution
• Query execution pipeline
• Result set propagation mechanisms
• Multi-database coordination

Understanding the engine internals helps when:

• Debugging complex concurrency issues
• Optimizing for extreme performance scenarios
• Building custom extensions
• Troubleshooting edge cases

Concurrency & Server Domain Management

Understanding Server Domain Attributes

All Dataset Module properties exist in the server domain, creating a shared resource environment:

Example Risk Scenario:
1. Client A sets: SQLStatement = "SELECT * FROM Orders WHERE Status='Open'"
2. Client B sets: SQLStatement = "SELECT * FROM Orders WHERE Status='Closed'"
3. Execute command runs with Client B's statement (last write wins)

Preventing Concurrency Conflicts

Strategy 1: Dedicated Query Objects

• Create separate query objects for different operations
• Assign unique queries to specific tasks or displays
• Avoid sharing query objects between concurrent processes

Strategy 2: Synchronization Patterns

• Use semaphores or locks in scripts
• Implement request queuing for shared resources
• Design state machines for complex operations

Strategy 3: Client-Side Processing

• Execute queries in scripts with local variables
• Process DataTables before assignment to tags
• Minimize server domain property modifications

Advanced Data Management Strategies

Concurrency Patterns

The Dataset Module provides three primary patterns for handling concurrent access:

Pattern 1: Direct Script Processing

csharp

DataTable result = @Dataset.Query.Query1.SelectCommand();
// Process data locally without server domain impact
foreach(DataRow row in result.Rows) {
    // Local processing
}

Pattern 2: Tag Distribution

csharp

// Assign to DataTable tag for module sharing
@Tag.MyDataTable = @Dataset.Query.Query1.SelectCommand();
// Now available to displays, reports, etc.

Pattern 3: Mapped Navigation

csharp

// Configure mapping, then navigate rows
@Dataset.Query.Query1.Select();
@Dataset.Query.Query1.Next(); // Moves to next row

Backup & Recovery

SQLite Backup Strategies

Option 1: Command Line Backup

bash

sqlite3 source.db ".backup backup.db"

• Simple and reliable
• Requires database file access
• Best for scheduled maintenance

Option 2: Online Backup API

• Backup while database is active
• Support for incremental backups
• Progress monitoring capability

Option 3: File System Copy

• Only when database is offline
• Fastest for large databases
• Requires downtime

Backup Best Practices

1. Schedule: Automate backups during low-activity periods
2. Verify: Test restore procedures monthly
3. Rotate: Maintain multiple backup generations
4. Secure: Store backups in separate physical location
5. Document: Maintain restore procedure documentation

Production Deployment Considerations

Design Principles

1. Isolation: Use dedicated query objects for different operations
2. Filtering: Always limit result sets with WHERE clauses
3. Security: Use parameterized queries exclusively
4. Monitoring: Track performance metrics and errors
5. Planning: Design for concurrent access from the start

Production Checklist

•  Before deploying to production:
•  Parameterized all dynamic queries
•  Implemented error handling for all operations
•  Tested concurrent access scenarios
•  Configured appropriate timeouts
•  Established backup procedures
•  Documented recovery processes
•  Verified timezone handling
•  Optimized query performance
•  Planned for data growth
•  Secured network access

Execution Process And Database Connections

The Dataset Module, specifically the TRunModule.exe (Dataset) process, reads all project configurations relevant to this module but does not directly interface with databases. The TServer provides database access services. The Dataset Module only uses TServer-provided services.

The execution of configurations made in Queries, Tables, and Files, which are Dataset features, runs in an independent process TRunModule.exe (Dataset). This process reads all the project configurations related to the Dataset Module. However, it does not directly execute the interface with the databases. Within TServer, there are services to access the database, and connections and calls to the database occur in TServer. The Dataset Module consumes this bank access service from the TServer.

Databases Connections

For each configured DB, only one access connection is created, meaning there's only one execution thread for each DB. If parallel execution of commands to the same database is required, you can create more than one DB pointing to the same database. The Dataset Module is a vital system component, functioning independently via TRunModule.exe (Dataset). Despite not interacting directly with databases, the Dataset Module reads project configurations and utilizes services within TServer to access databases. As a result, all database connections and calls occur in TServer.

Database Interactions

The Dataset Module provides two critical methods for executing data requests: 

• Synchronous
• Asynchronous

Understanding the differences between these methods, their default behaviors, and when to use each is crucial for optimizing performance and avoiding potential bottlenecks in your project.

Asynchronous Execution

This method is generally more performant and can help avoid potential locks. When properties in both Tables and Queries, such as the Select property, are triggered, the execution becomes Asynchronous. After the trigger, the execution usually continues. At the same time, the property is propagated from the trigger point, such as a screen or a script, to the server in parallel. It then synchronizes with the Dataset Module, which calls upon the necessary services from TServer to interface with the database. Upon TServer's return with the execution of the Select for the Dataset module, the result is updated in the properties and attributes of the module. Lastly, the mapping process is executed, updating Tags' values to the entire project.

Synchronous Execution

Certain methods have a Synchronous Execution. For example, when calling the SelectCommand method, the solution execution is paused until the Dataset Module carries out the service call to TServer, which interfaces with the database and returns the result to the Dataset Module. The Dataset Module then performs Tag mapping and returns the result, allowing the execution to resume.

Since the execution is paused, it's crucial to use synchronous calls carefully within mono-threaded code behind screens, whether WPF or HTML5. While synchronous execution can be appropriate in some instances, overuse can lead to performance issues. Mechanisms like the Script task or Action Server Request can facilitate asynchronous execution without using attributes. This is particularly useful when the result needs to be handled even after an asynchronous execution has been completed.

Client Vs. Server Tags in Mapping Results

In the Dataset Module, it's possible to utilize Tags from the client domain during Tag mapping. This capability provides additional flexibility when managing data in your project. When executing a synchronous or asynchronous command, the Dataset Module, not the TServer, carries out the tag assignment within the original domain of the call. 

If a call is initiated from a client, such as a screen or client script, the mapping operation between the received values and the tags occurs after TServer returns the result to the Dataset module. In this scenario, client domain Tags can be used in the mapping. The choice of the Tag domain is essential for controlling the scope of data availability. If you wish the data to be available project-wide, server domain Tags are the appropriate choice. However, if the data's scope should be confined to the client that initiated the call, client domain tags would be the best option.

This detailed understanding of the functionality of client and server domain tags in the Dataset Module's tag mapping process can enhance your project's data management efficiency and effectiveness. Selecting the correct tag domain based on your project's specific requirements is crucial, as it significantly impacts data accessibility and overall project performance.

In this section...