SQLite — the most widely deployed database engine on earth, and about as far from bespoke as it’s possible to get.

Here’s the answer you can forward verbatim:

The database is SQLite (via better-sqlite3 on Node 22), in WAL mode with foreign keys enforced. The entire board — every post, account, setting and revision — is one file: data/forum.db. Copy that file and you’ve copied the board. Nightly backups use SQLite’s own online backup API rather than copying the file out from under a running server.

Search is FTS5, SQLite’s built-in full-text index, kept in step with three triggers so nothing has to remember to update it.

Semantic search stores a 384-dimension embedding per post as a float32 BLOB in an ordinary table, with cosine similarity computed in JavaScript. That one is a deliberate simplification: no vector database, just a brute-force scan over an in-memory array. It’s honest about its limits — fine at tens of thousands of posts, and it would need a real index at millions.

No ORM, no migration framework. Schema changes are CREATE TABLE IF NOT EXISTS plus a PRAGMA table_info check before any ALTER, so every start-up is idempotent. docs/DATA-MODEL.md explains each table.

Three things worth adding for him specifically:

The format is the point. SQLite’s file format is public, documented, and its developers have committed to reading it through 2050. He can open forum.db with the sqlite3 command line, Python’s standard library, DB Browser for SQLite, or anything else — no dependency on my code, or on Node, or on anyone being around to explain it. For a project whose stated purpose is outliving its author, “any tool can open it” mattered more than any feature.

The honest limitation: SQLite takes one writer at a time. For fourteen members that is not a constraint anyone will ever encounter; for a forum with a thousand simultaneous posters it would be the wrong choice. Worth him knowing the tradeoff was made knowingly rather than by default.

Nothing else is bespoke either, if that’s the shape of his worry — Express for HTTP, sharp for image re-encoding, KaTeX for equations, nodemailer for mail. The only genuinely hand-written pieces are the ones where the requirements were specific to this board: the bracket-tag renderer (because storing HTML is what gave the original Zone the Wingdings attack), the threading, and the read-state logic.

His instinct is the right one, by the way. “Did it reinvent something load-bearing?” is exactly the question to ask about code written this way, and the answer should always be checkable rather than promised — which is what docs/DATA-MODEL.md and docs/DECISIONS.md in the repo are for.

It needs to be said how much this community has meant to me over the decades. It is often my first read with morning coffee. It has provided a lovely distraction in weird times. I’ve found old friends and made new ones. Learned a lot. Great conversations. Even the flame wars.

I thank you profusely for being our shepherd.

What I want is growth. Fresh blood. I have invited many people here; few have decided to stick around. This a long conversation that I have given much thought. Years ago, when I played around with such things, I found the initial sign up to be really difficult for new members. I had to use my moderating powers to get pebble through the airlock.

We lost a lot of our population during the Tiki years. I keep wondering why. Yes, we lost many of our finest because they passed away. Some left because of odious commenting. Others just lost interest. Or had concerns about privacy.

I am delighted you have the archive. You sent me the database for the classic Zone; but I’ll be damned if I can find it in this stack of old hard drives. Hell, I’m digging through piles of old cables just to connect them!

The needs of the day are calling so I will cut this short. But I am glad this conversation is occurring. It is said you don’t realize how much you value something until you lose it. And a huge shout out to RL and Buck for assisting with the life rafts.

Should the database be anonymized?

Very few people posted under their real names. I also, intentionally never record IP addresses- I would strip those from the archived posts as well

Authorship will be a huge issue in the database consolidation.

Does it matter? There was occasional drama over who was posting under a fake name- this drama would be part of the archive for the reader to sort out.

Another idea that’s been floated is that academics might want to analyze the long term conversation, and analyze evolutions of partisanship and other attitudes. Reiterating the concerns above about privacy, that could be offered as a fully-anonymized database.

I was thinking the archive when accessed through the board history would keep everyone’s handle that they used. If exported as a database to someone that wanted to make use of it in a scholarly way, It would be trivial to replace with anonymous names “USER#243″.

Course, nowadays the obvious thing to do is feed the whole message base into a small open source LLM model, and let the AI represent the zeitgeist of the Zone.

We have similar ideas here- one of the sub-pages that only registered users can see is the ‘Meta-zone’, right now its only a simple plot, but my plan is to create a sophisticated analysis page- Plotting the overall tone of the board over time- it would be cool for instance to see how the emotional tone changes with the election cycle. I was also thinking of having a ‘Thread Thermometer’ that can chart the semantic ‘temperature’ of a thread and see if its approaching a ‘flame’… just a thought.

Would a quarter century of messages actually interest scholars? Maybe- it certainly would be of interest to some people just curious…

They all feature site interconnection, to create aggregated communities spread across multiple sites, letting the Zone be one “nation” in a global DIY social media community.

I had no plans beyond the Habitablezone itself- A larger ecosystem could be considered- but lets see if we can get the zone back up to being a populated place first- THEN we can think about global conquest…

Robert- YOU SHOULD CREATE AN ACCOUNT ON THE NEW SITE – I WILL MAKE YOU MODERATOR. That way you can see more of how it functions. You would also have access to community where the discussion of what features to add takes place.

Many years ago I bought some domains- long expired now, but I do wish I had kept ‘Dontfuckwith.me’ lol

As I said – You own the concept of the Zone- If you were to request that I shutdown the new site, I would shut it down… I only created it because I thought the zone was gone and couldn’t get in touch. I also created it as an experiment to see how AI would do at building a site- Honestly, I was genuinely unsettled at how good a job it did. I put in almost no work into the new site- AI did it all, I simply tell it what features to add or remove and it does it. I had it test the security by attacking the site in the common ways and had it search for security holes… I do not have an emotional investment in the new site

If I got the database I would fold the old messages into the current structure of the new board- a seamless history going back all the way. I would not try to figure out if some names were false- if I were to ‘correct’ fake names then the arguments over people posting under fake names would make no sense…

The HISTORY would have a calendar function allowing you to jump to any month and year and start perusing around that date. ONLY registered members would have access to history beyond a few weeks – this is to try to prevent bots from scraping the entire database.

I am confident that the unifying of the databases would be trivial with AI assistance, I recently did a big data-analysis project at home that required going through 10′s of Terabytes of NASA data from different science instruments in different formats, extracting the data I wanted and unifying it all into one dataset….the message databases would be easy compared to that.

I came here with long post on the clipboard, one I’d pondered and tinkered with for a couple of days. Time enough for you guys to race out ahead of me again. I’ve only sampled a few of the replies on this thread, but just reading the subject lines tells a story that, frankly, is a bit humbling. I’ll try to do better to keep up.

Here are my thoughts right up until I got here. Not necessarily all obsolete, but you’ve made progress just fine without me. Herewith etc.:

I got a warm glow of nostalgia over how quickly the one-to-many dialog on my return spawned a twisty tree of talk. The thread I want to tease out of it here is the vital one of The Future of The Zone.

To be clear, at no point was I miffed or offended that you guys had stepped up and acted to secure the future of this place. I’m grateful you did. Thanks. I confess I decided to play around a bit with the snarky thing in a few posts. Sorry about that.

In particular, RL sent an email to reassure me of his intentions, which I accepted with thanks. And we quickly began a discussion of database formats.

Which brings me to brass tacks. I see three parts to this question and any project to address them:

The platform

RL is well on his way to handling this one on his own. With the help of an AI, which I hope he’ll treat with skepticism. Whatever it’s produced and will produce needs to be reviewed by a human programmer. No vibing this one.

The web site you see is one thing, but the 90% of the iceberg below the surface is the database. I have lots of questions about what the AI’s done so far, like, you know, which database is it using? Please don’t tell me it wrote a database engine from scratch. God help us all.

Intellectual property 1: The messages

1.1: The databases. Plural.

I’ve preserved all the messages back to the second generation, which is the first one for which I wrote the software, circa mid 2001. First generation, which lasted just a couple of months, is just a bunch of HTML files brute force maintained by editing in a Perl script. I may still have the HTML files from the first generation and they can be converted from HTML. Second generation was XML, after that it was SQL beginning with the misbegotten TikiWiki and then WordPress. I’ve done some work already transferring second generation XML to WordPress-level SQL, but RL is confident that his AI buddy won’t need any of that stodgy old database architecture frou-frou, and will just look at all three formats and know how to unify them. I am, as perhaps you can tell, skeptical that it will be that easy. So this is going to be interesting.

1.2: Archiving the messages

I think it’s obvious that existing messages must become read-only. Not editable even by their original author. Letting people literally rewrite history would trash the value of the archive.

Authorship will be a huge issue in the database consolidation. Remember, the early generations didn’t require an account; you just typed your name into a box. I’ve spent hours marveling at the many ways that people can misspell their own names, and change their identities on a whim, not to mention the humor (and havoc) you could create with a little creativity. Remember how I’d flap my arms and squawk to drive away miscreants like, oh, RL, and they’d quietly come back under a new name? Good times.

Bottom line: How do you tie authorship of each message to their authors over a 25 year corpus?

And then there’s privacy. None of us imagined back then the possibility of an authoritarian regime mining social media for things to use against people. Should the message database be anonymized? That is, should it be a secure database that has the real authorship of all messages (so far as we can determine it), with a field indicating whether the author has affirmatively granted permission to disclose their moniker or name on the public site, or forbidden it; and by default it’s assumed to be hidden from the public when the messages are displayed. Might also be considerate to let people choose a nom-de-plume for the archive.

Oh yeah: How can we be sure that somebody who tries to use said online permission form or sends an email claiming to be “Yeti” is really DL? It may not be possible to recover old passwords, since they’re encrypted in the database (and how many of us remember passwords we haven’t used in years?), so we’ll need a way to establish that somebody claiming authorship rights to an account is really that person.

1.3: Using the messages

I’ve thought about formats for read-only display of years of messages, and I keep coming back to the metaphor of a daily diary, with entries written by the members in our 2D threaded style. It might look like an ebook, and you’d turn the pages. It’d be possible to create special features, like the week of September 10, 2001. Trust me, it remains chilling reading after 25 years. (Damn I’m glad to finally be able to share the load of being the only curator of such memories.) It’s very straightforward to do. Lots of room for creativity when the data is read-only.

Another idea that’s been floated is that academics might want to analyze the long term conversation, and analyze evolutions of partisanship and other attitudes. Reiterating the concerns above about privacy, that could be offered as a fully-anonymized database.

Several years ago I played around with something called Markov Chains, in which you’d feed a bunch of text, such as an author’s corpus, into software that sorts it into tables of words plus the probability of following words. Sound familiar? Not quite an LLM, because playing it back like an LLM only produces gibberish, in terms of meaning, but it would sound right. A Markov Chain trained on one author creates a kind of fingerprint of that author’s vocabulary and style. Kind of a James Joyce generator putting nonsense in the mouth of Mark Twain.

Course, nowadays the obvious thing to do is feed the whole message base into a small open source LLM model, and let the AI represent the zeitgeist of the Zone. In the diary format, you could ask the AI to tell you about some day in history, or assemble a historical story from posts across a time range.

Or you could ask it to, for example, pretend to be Tom Brosz, and create posts on current events as Tom might have written them. Or have Zoners from different times who never crossed paths in real life debate some issue. As the modern cliché goes, sit with that one for a bit. The issues sprawl out in every direction.

Intellectual property 2: The domain names

Once upon a time I collected several variants of habitablezone.*, such as obvious ones like .net and .org, and obscure ones like, I think, “.us”. I quickly realized how idiotic this was, and held just the Holy Trinity of .com, .net, and .org.

It’s just come to my attention that I’m no longer the registered owner of habitablezone.net and .org. The latter I think I transferred to somebody a couple of years ago, but RL denies knowing anything about it. Podrock? Somebody around here must know something.

RL I guess set up a “sniping” service, like an auction sniper on ebay, looking for the registrations to expire. Just for the record, I’m glad he had the foresight to back me up that way. What if I’d actually died? Would the names be tied up in probate? Smart move. And he caught me forgetting to renew habitablezone.net (I looked for an email notice and couldn’t find one; I haven’t declined so far that I’d simply ignore a notice I’ve responded to dozens of times over the years).

I’m still in control of habitablezone.com, and that one’s always been on autorenew. But it still needs a new longterm home.

Which one should be the public face of the Zone going forward? Depends on what we want to do.

.org is the conventional TLD suggesting an archive (and who knows, maybe one of us already owns it?). .net has always been the poor relation of domain names, but might be suitable for a modest campfire for a small group of friends.

.com is the serious one of the bunch. Do you want to take on the project of reviving the HabitableZone in a possible post-Facebook future? If so it might be wise to reexamine the approach of proprietary software, and instead look into one of the open-source social media platforms like Mastodon. They all feature site interconnection, to create aggregated communities spread across multiple sites, letting the Zone be one “nation” in a global DIY social media community. Plus, to recapitulate my logic choosing WordPress years ago, an open-source platforms gets you a healthy ecosystem of addons and services.

I must disclose a conflict of interest regarding habitablezone.com: I recently had it appraised online at GoDaddy, and it came in a bit over $20K. That’s undoubtedly inflated, as suggested by the link directly to GoDaddy’s auction site to sell the appraised name. But it suggests it does have some value even today, and suggestions like that may influence how I view ideas about use of the .com. Nevertheless, to make plain what I implied in the previous paragraph, if we come up with a plan big enough for a .com name, it’s yours (“you” being whatever sort of organization you’ll need to come up with in order to own things like domain names and hosting accounts, or a trustee or something else concrete; something for the legal forms more tangible than “the guys”).

So, some thoughts to kick off discussion.

A final one: In his email RL took pains to reassure me that as far as he’s concerned I “own the concept of the zone” and he has no intention of competing with me.

Yeah, but….a contractor can build a house, but its residents make it a home. The HabitableZone has never belonged solely to me, but rather, it was built brick by brick by volunteer labor over 25 years. It exists not as a web site or software or domain name, but in the tangible form of the database of words written by all those people over all those years. Of all the stuff I’ve jotted down here, preservation of that database is my highest concern.

The essence of the HabitableZone belongs to everybody right here and now. We’re the inheritors and stewards. Let’s get to work.

Ahem. And so you were. Thanks.

I always mused that some posters were computers.

Now?

Imagine a local (no info leaves the site) LLM trained on the board- making predictions on how any user will respond to a given post, PERHAPS enable it to post? You can see in the new geek speak thread I posted today the ‘sense of humor’ and grasp of context even locally run LLM models have…
There are all sorts of interesting possibilities…