I build and run AI systems end to end — and I measure whether they work.

I wanted to know whether I could put a working assistant on a live phone call, end to end, by myself.

It listens through speech recognition on the Mac, thinks with Claude, speaks with ElevenLabs, and dials out through my iPhone with one click.

Sensitive numbers never reach the model. They sit in a separate vault the assistant can use but cannot read, because a model that can see a card number can also repeat it. That was a design decision I made before writing the call logic, not a patch afterwards.

I covered it with 174 automated tests. Not for ceremony — voice systems fail in ways you don't catch by hand. A timeout, a half-heard digit, a call that ends mid-sentence. Those only show up if something is checking every time.

What I'd do differently: I built the conversation layer before the failure handling, so early calls broke in ways I had to go back and design around. Next time I'd write the failure cases first, because on a phone call there's no retry button for the person on the other end.

I want AI agents helping with my mail. I don't want them seeing my card numbers.

So I built a local service that sits between the mailbox and any assistant. It masks card numbers, Social Security numbers, bank details, one-time codes and API keys before a model sees the text.

It's read-only by construction. It cannot send, draft or delete. That matters more than it sounds: it means the worst thing a bug can do is fail to mask something, rather than destroy mail or reply to someone as me. I'd rather cap the damage than trust the model to behave.

This isn't a demo project. It's the reason one of my agents is allowed into that mailbox at all. Without it I wouldn't have given anything access, and the work I've built on top of it wouldn't exist.

What I'd do next: the masking is pattern-based, so it catches the formats I thought of. I want to measure what it misses on real mail rather than assume, because "no false negatives yet" is not the same as "none."

I publish a children's book through winterfallbooks.com. It has one sale and one review. That's the honest starting point, and it's what makes it a useful thing to test on — there's nowhere to hide behind existing traffic.

The question is narrow: does a free-story email offer convert better than a different headline on the same page?

I wrote the test down before collecting anything. One primary metric, a minimum sample per arm, and a stopping rule: it reads out at the first of 200 visitors per arm or eight weeks after launch, with a fourteen-day minimum. I decided what each outcome would mean before any data existed, because a test you design after seeing the result isn't a test.

The funnel that reads it is already built — views, clicks, sign-ups, purchase clicks and sales, per variant — and the stopping rule lives in it rather than in my head. There's no chart yet because there's no data yet, and I'm not going to show you an empty one.

Traffic is near zero so far. The likely outcome is that this runs to the time cap and reports underpowered: at 200 visitors per arm, only a large difference would show clearly. If that's what happens I'll say so, with the sample size and what I'd change.

I make long documentary episodes for Drive and Listen — history, money and technology, built for listening on the drive — and I didn't want a factual mistake reaching anyone. They're audio first, with simple visuals to keep production light. The two hosts are AI voices; I write and research every episode, and the channel says so.

So I built a pipeline that runs from primary-source research and fact checks to a two-host script, synthetic voices, video, metadata and a private upload. Episodes upload privately, and the rule is that none goes public until I've watched the full private video and approved it. Every episode lists its key sources in its description, so the fact-checking is something you can check rather than something I claim.

I added several checks after specific failures. A made-up quote reached the audio, so I now require a primary source for every factual claim and a word-for-word check of every quotation. A voice sample dropped a spoken year, so I now require the audio to be checked against the script word by word before it reaches me.

What I measured: one September re-render of the video for a 40-minute-42-second episode took 482 seconds, about 5.1 times playback speed. It changed only visual cues, so all 163 voice lines came from cache. One audio build's voices cost $0.86 across 26 requests. That's the voices for that one build, not the whole production.

The limit: the checks don't guarantee that a factual error can't get through.

I use AI models to check other models' work: fact checks, code reviews and audits. I wanted reviewers that open the sources instead of judging how the writing sounds. And a review that finishes without an error hasn't necessarily reviewed anything.

So a reviewer earns its place one kind of job at a time. A model can be admitted for one kind of review and kept out of another. Web research is a separate lane. Its output is treated as untrusted, and a result with no sign that a search actually ran is rejected.

Testing kept one configuration out of fact-checking. On its last retest, in late September, its runs finished without errors and quickly. But one review's output couldn't be parsed, and a phrase it quoted as the source's words wasn't verbatim. A clean finish would have looked like a pass. Checking the output itself is what caught both problems.

The limit: a completed review still needs its claims and quotations checked against the sources.

I wanted AI agents doing real work — research, writing, data — without two things going wrong. Losing track of what each one had done, and quietly giving them more access than the job needed.

So the agents are separate, and deliberately unequal. About fourteen run on my own machines, each scoped to one job. The agent with access to my mailbox has no shell and can't reach my credentials. I tested that directly instead of assuming it, because an untested boundary isn't a boundary.

One agent's only job is cost. It compares how much of the week's planned work is done against how much of the week's AI budget is spent, and defers work only when the budget runs ahead of the work. Everything needing a decision lands on one short list; no agent approves anything for me.

What I measured: I gave five models the same research task and checked, on each model's own links, how many of its confident claims held up. One held 12 of 14, the next 9 of 18, and the other three held 0 of 9, 0 of 13 and 0 of 25. None was shown to be made up. Most misses pointed to a general page instead of the specific item, or to one that had closed; a few pages couldn't be read, and a few were real but didn't fit. That is one task under one check, not a verdict on any model — but it decided which one I kept.

Then a daily check reported seven records as closed. Three weren't: the listing had expired, not the thing itself. So it verifies against the source now, not the copy.

Task mirror. A one-way copy of my decision list and the agents' work status reaches the task app on my phone. Their records stay authoritative; it writes only to its own lists and has no method to delete a task.

Email intake. A separate mailbox takes messages I forward to it, accepted only from an approved sender whose domain passes an authentication check. Message text is treated as content, never as instructions. The part that reads that mailbox has no code to send or delete mail.

They run. I still decide.