Live reloading of tiddler files

[Jermolene]

With this PR TiddlyWiki on the server gains the ability to scan tiddler directories for dynamic changes, and automatically reload the changed files.

This is a long standing request. @natecain proposed #176 in October 2013 to address it (it was merged but later reverted after problems were reported). There is also a feature request in #1530 from 2015. UPDATE 18/4/26: I missed #3060 too.

This PR adds a new dynamicStore: { saveFilter, watch, debounce } property to tiddlywiki.files. A dynamic store is a folder that is loaded at boot, watched live with chokidar, and then used as the save destination for tiddlers matching a configurable filter.

The new dynamicStore optional property that marks the directory as a //dynamic store// that is both loaded at boot and actively watched on disk. The filesystem syncadaptor uses the watcher to pick up out-of-band changes (e.g. edits made by an external editor) and folds them into the running wiki. The object has the following properties:

• saveFilter - (optional) a filter evaluated against each tiddler the wiki tries to save. Tiddlers that match are saved back into this directory instead of the default tiddlers folder. The first matching dynamic store wins, so specificity matters when multiple stores are registered.
• watch - (optional, defaults to true) set to false to disable the chokidar watcher for this store (tiddlers are still loaded and save-routed, but external changes are not picked up live)
• debounce - (optional, defaults to 400) the per-file debounce window in milliseconds. File events that occur within this window of a previous event for the same file are coalesced, which avoids duplicated reloads during atomic-rename saves performed by many editors.
  Changes to a file on disk are diffed against the tiddler currently in the wiki before being reported, so self-writes performed by TiddlyWiki itself do not trigger spurious reload events. JavaScript module tiddlers (tiddlers with type: application/javascript and a module-type field) are never hot-reloaded, because that would require restarting the TiddlyWiki process.

This example treats a sibling content/ folder as a dynamic store: every .md file is loaded at boot, every markdown tiddler saved by the wiki is routed back into content/, and external edits to those files are picked up automatically by chokidar.

{
    "directories": [
        {
            "path": "../content",
            "filesRegExp": "^.*\\.md$",
            "isTiddlerFile": true,
            "searchSubdirectories": true,
            "dynamicStore": {
                "saveFilter": "[type[text/x-markdown]]",
                "watch": true,
                "debounce": 400
            },
            "fields": {}
        }
    ]
}

[netlify]

❌ Deploy Preview for tiddlywiki-previews failed. Why did it fail? →

Name	Link
🔨 Latest commit	9830d43
🔍 Latest deploy log	https://app.netlify.com/projects/tiddlywiki-previews/deploys/69e49533c4cf2a00082e9416

[github-actions]

Confirmed: Jermolene has already signed the Contributor License Agreement (see contributing.md)

[github-actions]

📊 Build Size Comparison: empty.html

Branch	Size
Base (master)	2489.5 KB
PR	2492.2 KB

Diff: ⬆️ Increase: +2.8 KB

---

✅ Change Note Status

All change notes are properly formatted and validated!

📝 $:/changenotes/5.5.0/#9806

Type: feature | Category: nodejs
Release: 5.5.0

Filesystem dynamic stores with live reload via chokidar

🔗 #9806

👥 Contributors: Jermolene

---

📖 Change Note Guidelines

Change notes help track and communicate changes effectively. See the full documentation for details.

[saqimtiaz]

@Jermolene I last worked on this in late 2024 and was able to get it working using fs.watch directly, avoiding the chokidar dependency, with the caveat that there were some false positives since fs.watch triggers more than once for every file change. I will see if I can dig up the code before I head out on the road for a couple of months.

[Jermolene]

@Jermolene I last worked on this in late 2024 and was able to get it working using fs.watch directly, avoiding the chokidar dependency, with the caveat that there were some false positives since fs.watch triggers more than once for every file change. I will see if I can dig up the code before I head out on the road for a couple of months.

Thanks, I do recall your work on this. Avoiding the dependency is certainly desirable.

[pmario]

Did you test this on a Windows machine? From time to time I do experiment with file watchers. Most of the time Windows Explorer completely kills the experience. It is able to lock files, and keeps them locked, in a way that completely confuses the watcher processes.

So on Windows, we need to test that very carefully.

[Jermolene]

Did you test this on a Windows machine? From time to time I do experiment with file watchers. Most of the time Windows Explorer completely kills the experience. It is able to lock files, and keeps them locked, in a way that completely confuses the watcher processes.

So on Windows, we need to test that very carefully.

That was my reasoning for starting with chokidar. It is widely used and battle tested. If we rolled our own alternative I think we’d struggle to match their test suite.

[linonetwo]

I have an working example at https://github.com/tiddly-gittly/TidGi-Desktop/blob/master/src/services/wiki/plugin/watchFileSystemAdaptor/WatchFileSystemAdaptor.ts
(And https://github.com/tiddly-gittly/TidGi-Desktop/blob/master/src/services/wiki/plugin/watchFileSystemAdaptor/FileSystemWatcher.ts )
I already use it for a while, when using it I find several dangerous situation:

1. when user typing, saved change may trigger file watch, so it echo back to frontend, cause new change gets overwritten. I use lastWrite to record self write time and size to avoid this. There is a doc https://github.com/tiddly-gittly/TidGi-Desktop/blob/master/docs/internal/IPCSyncAdaptorAndFSAdaptor.md
2. chokdar will use JS thread, and can't handle rename, I switch to nsfw after deal with it for several times (but it requires native file, I shared about this before Reload changed tiddlers from disc  #3060 (comment) )
3. sometimes .meta change or delete, need to map it to the tiddler.
4. Sub-wiki (you can split private content to another folder, and still have main-wiki opensource in Github to publish as blog) also need to be watched and map to main-wiki
5. $__StoryList sometimes create annoying $__StoryList_1, need to ignore this
6. git checkout will bring in too many change, git revert/checkout needs delayed deletion processing

This will require many test case, If core provides this feature, I can finally not maintaining this by myself. I was planning to let copilot migrate it from my code to the core, but chokidar dependency needs your approval, and I haven't feel it being stable in many edge case, so I haven't dare to do so. I sometimes afraid to enable this feature, afraid of typed content lost by echo.

[Jermolene]

1. when user typing, saved change may trigger file watch, so it echo back to frontend, cause new change gets overwritten. I use lastWrite to record self write time and size to avoid this. There is a doc https://github.com/tiddly-gittly/TidGi-Desktop/blob/master/docs/internal/IPCSyncAdaptorAndFSAdaptor.md
2. chokdar will use JS thread, and can't handle rename, I switch to nsfw after deal with it for several times (but it requires native file, I shared about this before Reload changed tiddlers from disc  #3060 (comment) )
3. sometimes .meta change or delete, need to map it to the tiddler.
4. Sub-wiki (you can split private content to another folder, and still have main-wiki opensource in Github to publish as blog) also need to be watched and map to main-wiki
5. $__StoryList sometimes create annoying $__StoryList_1, need to ignore this
6. git checkout will bring in too many change, git revert/checkout needs delayed deletion processing

Thanks @linonetwo we should consolidate what we can into this PR.

Note that (4) is already supported in this PR using dynamic store save filter.

This will require many test case, If core provides this feature, I can finally not maintaining this by myself. I was planning to let copilot migrate it from my code to the core, but chokidar dependency needs your approval, and I haven't feel it being stable in many edge case, so I haven't dare to do so. I sometimes afraid to enable this feature, afraid of typed content lost by echo.

We’ve held off implementing this feature before because of concerns like this. I think now it is essential that we offer this feature as soon as possible in order to be fungible with Obsidian for applications like LLM Wiki.

I share your concerns about lost data. I would like to explore ways of implementing “running backups”, for example prior to deleting or overwriting files we copy/move them to a backup directory. It would echo one of the features I like about the single file edition of TW: that one can keep bashing the “save” button to make continuous backups as one works.

[pmario]

@Jermolene .. I do not know if you did realise, that my tw-mcp server plugin is able to connect several llm clients at the same time to a single TW server. They all share the same $tw.wiki store.

Most of the mcp-tools have direct access to the server $tw.wiki store and can manipulate it. If they do so the file syncer automatically syncs the info back to the filesystem.

If files are written to the filesystem by an external editor. That's not detected.
But the mcp server has a tool: reload_tiddlers that will reload all files from the filesystem without the need to re-start the server.

There are some exceptions. system tiddlers still need a server restart. ...

I did not read the code from your plugin in detail. So I do not know if this info is actually useful. I only wanted to let you know.