An update on 5.x

[benkeen]

Overview

Since late November, I’ve stopped doing so much voluntary overtime at my job at Microsoft and have been focusing my spare time on this project instead. Last year saw a huge reduction in users for the site, which I attribute partly to the fact that it hadn’t been updated and had gotten stale, but mostly to the rise of AI. If people need random data for any purpose, they can now just ask AI and get a tailored response back. In many - even most - cases, that solves their needs far more easily than hunting down a script online and hoping it does what they want.

But AI is now the reality we all live in, so for v5 I’m paving the way to unblock the CLI portion of the app, letting agents leverage this tool under the hood to generate data. That seems like a reasonable path forward for the script to keep it relevant and useful.

Main goals of 5.x

There are two main goals.

1. Proper documentation and making it easier to run the app locally

Up through v4, I think I did a pretty good job documenting the script. Installation was easy - it was PHP & MySQL, and could be run with MAMP/LAMP/WAMP. But I really wanted to move past PHP in particular, so with v5 I moved all data generation to the client with web workers, and switched to a Node/Express backend with Docker. Technology‑wise I was much happier, but I completely failed to properly document the script. Installation became tougher, and that’s something I absolutely want to fix with v5.

2. Unblock adding the CLI

The CLI will give you access to all of generatedata’s functionality, but via the command line. This opens up enormous possibilities for generating data sets programmatically. Saying this feature is overdue is the understatement of the century... sheesh.

I got a first draft of the CLI working over three years ago, but I had too many misgivings about the code. I was cramming a square peg into a round hole, and it was too shaky a foundation to release as-is. Unfortunately, with my day job, I didn’t have the energy to do another full revamp at the time, so I put it off.

So what 5.x does is re‑architect the whole app to:

• move the code to a monorepo (Turborepo), separating each part into logical, discrete packages
• mature the backend server code: move it to TypeScript and, above all, provide proper typings for everything - especially plugins, which will be vital for the CLI and input validation.
• update all dependencies and technologies, which were a few years out of date.

Once these are done, creating a standalone, publishable @generatedata/cli npm package will be significantly easier.

Where we’re at (Jan 11th, 2025)

All the big problems are now solved: the major architectural pieces are done, and the app boots up nicely in local dev mode via a single command. The server and client apps have been separated and isolated, and the library packages are in place. The web worker generation process, especially for dev, is still cumbersome, but I may live with that - I'll just document the limitations.

There’s still a lot of cleanup, refactoring, and bug fixing to do, but we’re in very good shape. I’m still hoping to create a truly standalone Docker image for the app that won’t require Node, npm etc. locally. I’m not 100% sure it’s doable, but we’ll see. I should know in the coming days.

Oh - and in terms of order, my plan is:

1. Complete the open‑source app
2. update the website
3. move on to the CLI

So that’s the update! The luxury of open source is that I don’t have to provide estimates - it gets done when it gets done. :) But I’m picturing the initial 5.x release in the next few weeks.