John Stockton's Home Project Wiki
Challenging technology projects for home
Found: 2 Entries
Dec 1, 2010 Format for Print
1. tFM Database Overview

This is a summary page for my tiny File Manager & database project called tFM. The idea is to make something that resembles FileMaker Pro, but have it totally focused on the web for every aspect of its user interface. The goal is to make it easy to use and not require learning complex SQL queries, but still powerful enough to make most dynamic pages that you would find on the web.

Some of the major features include:

  • Tags that can be inserted into web pages for dynamic content
  • Tags that can cause database search/edit/delete operations
  • Schema abstraction beyond rows & columns to include collections and files
  • User Token and Variable support for intra-page calculations
  • Simple embedded scripting support (still a work in progress)
  • Simple triggers based on data operations and timer values
Other system level features include:
  • SMTP email send support
  • Yahoo quote retrieval support
  • Database replication which operates on both LAN and WAN (push & pull records)
  • Monitor function that restarts a server instance if it becomes unresponsive
The current state of the project is still partially complete, but some core functions such as the web server and database have been stable and in continuous operation for the past two years (my blog site and our neighborhood website have been hosted on tFM since 2008).

Since this site has been available, I've used it to document my thoughts about implementation trade-offs where I'm currently focusing my spare time efforts. The following are some of the recent areas of development:
The current implementation is limited to Windows, but the next generation of the program will be ported to Mono Develop (.Net) for portability to Linux and the Mac OS.
Category: TFM 1.00 Submitted by: John
Dec 10, 2010 Format for Print
2. tFM Lazy Replication
  Implementation Notes
System Requirements:

Introduction: The idea of a distributed database system is to have multiple copies of a database spread out on multiple computers, some geographically diverse. The general requirements are to be able to tolerate a "seldom connected" scenario, synchronize over both a LAN and a WAN, have relatively low overhead during the synchronization operation, allow for edits to take place on any copy of the database but resolve conflicts either during or after synchronization, and finally allow for deletion to take place while databases are not connected.

Sync vs. Async: At the highest level, synchronization of databases can be done synchronously or asynchronously. For applications such as transaction processing where workloads may be split among multiple databases, there is a significant need for synchronous replication. These systems typically have very high performance interconnect and are always online and available. This is not the application that tFM is targeted to perform since there are plenty of large, well established solutions in this space and no one needs another.

Asynchronous replication is easier to implement, but has the shortcomings that data can be stale or worse, edited with difference contents in two different locations. This requires that changes be logged and either not committed until all databases are ready, or be able to be undone with some reasonable conflict resolution process. The application scenario that tFM is targeted at is where one copy of the database might be on a laptop and not connected to a network at all. Edits may be performed on the laptop while away from a network, but as soon as it is connected to the network a synchronization process should be started to make the databases consistent. If inconsistencies are found, they can be flagged and resolved at a later time. Further edits to inconsistent records should generate an alert, but may continue as the user will have to decide which edits to keep/overwrite later.

Triggers vs. Change Logs: The two common ways to implement synchronization are triggers that detect when a record is being edited and invoke a synchronous change process, or change logs that keep track of what has changed and allow synchronization to occur later. The choice of asynchronous replication forces change logs to be implemented. These should be maintained until all peer databases are consistent, then they can be deleted. In the current tFM implementation, logs are kept only after databases have been initially synchronization.

Record Edits:

Record Ownership:

Record Deletion: One of the difficult things to implement with lazy database synchronization is record deletion. The first implementation of it wound up with every record that was deleted being put back from the other linked database. The current thinking about how to do deletions is to effectively implement a two-phase delete operation where in the first phase the record contents are deleted, but the RECID and TSTAMP fields stay around


Implementation Notes:

Initial Synchronization Phases:

  1. Top level database comparison
  2. Record level comparisons
  3. Local Record Newer
  4. Remote Record Newer
  5. Flag deletions


Category: TFM 2.00 Submitted by: John
Home Projects
About this Wiki

This wiki is for the development of a few home projects. The idea of a wiki is a place where it is easy to post and edit documents on the web. Anyone can read the documents, but posting and editing is limited to those authors that have password access.

Category Search
  • Autonomous Helicopter
  • CNC Mill Project
  • tFM File Manager Project
    Contact Information

    John Stockton
      4233 Hidden Canyon Cove
      Austin, TX 78746-1256