Same Question, Different Answer: How I Make AI Workflows Hold Still

Ask the same model the same question twice and you can get two different answers. That is called non-deterministic behavior, and it is the reason AI pilots stall when they become workflows. You are right about the fix. Deterministic process around the model is what holds it still.

A short thread in my feed put it plainly. Ask the same AI the same question, get a different result. The reply underneath said the same thing I tell clients. It means you get a new answer every time, and it is one of the biggest issues with AI when you try to use it for workflows. It does the task different each time.

That matches what I see in rescue work. A demo passes once. The same prompt fails on Tuesday. The team blames the model. The real problem sits one layer out. An LLM call is a statistical dependency, not a pure function. If you want the same job done the same way every time, you do not get it by asking nicer. You get it by wrapping the call in code that checks shape, checks facts, retries on purpose, and stops on budget.

What non-deterministic means in practice

Non-deterministic means identical input can produce different output across runs. For chat that feels like variety. For workflows that feels like breakage. A refund ticket routes to billing on Monday and fraud on Tuesday. A report includes the table once and drops it the next run. A deploy step runs in a different order with different shortcuts taken.

I see this pattern in deterministic versus stochastic fleets: drafting copy wants variance, closing the books must not have it. The fix starts by naming which mode each workflow lives in, then building the rails for the strict ones.

Why temperature 0 does not save you

Setting temperature to 0 makes sampling greedy. It always picks the top token. That part is deterministic. The scores feeding that choice are not stable on a shared endpoint.

A DEV deep dive from September 2026 shows the failure clearly. One engineer looped a byte identical request 500 times with temperature 0 and a pinned seed. Twelve responses came back different. One classified a refund ticket as billing instead of fraud. Same prompt, same snapshot, real behavior change.

The cause is arithmetic plus scheduling. Floating point addition is not associative, so grouping changes last bit rounding. GPU kernels split reductions across blocks based on tensor shape. Shape depends on batch size. Batch size on a hosted API depends on who else hit the server that millisecond. Your request at 3pm lands in a batch of 48. The same request at 4am lands in a batch of 3. Different tiling, different reduction order, slightly different scores. Most positions have a wide gap between top token and runner up, so nothing flips. A small share are near ties. There the top two gap is smaller than the numeric noise, argmax flips, and autoregressive decoding amplifies one token into a different second half. Ambiguous inputs fail most because near ties live there.

Seed helps with sampler randomness. It does not pin the math. Top p at 0 does not fix it either. Retry until match is rejection sampling with a bigger bill, not determinism. Bitwise repeatability lives on your own box at batch size 1 with pinned libraries and drivers, or with batch invariant kernels you have confirmed in writing. Everywhere else, design for a measured flip rate.

You are right, with one sharper edge

Your instinct is correct. The answer is deterministic process around the call. I would sharpen it this way. Do not try to make the model deterministic. Make the system deterministic about what it accepts, what it retries, and when it stops.

In my experience from 50 plus builds, the harness ends up as 80 to 90 percent of production code. Input construction, schema, tool permissions, state, checks, routing, budgets, escalation. The model call is the small part in the middle. That matches the harness engineering writeups from 2026: the intake format, repo context, permissions, execution state, checks, and escalation rules move work from request to merged code. The model proposes. The code disposes.

This is also the shift I describe in the move from prompting to loop engineering. One good prompt helps once. A loop with checks helps every run.

The five parts I put around every strict workflow

First, I shrink the output space. A classifier that emits one enum token has far fewer near tie positions than a paragraph that buries the label in prose. Enums for anything that drives branching. Priority low, medium, or high. Intent refund, technical support, sales, or other. No creative values.

Second, I enforce schema at decode time. Native structured output modes now exist across providers. OpenAI response format with json schema strict true. Claude output config format on current Claude models. Gemini response mime type application json plus response json schema. AWS Bedrock structured outputs through the Converse API. These use constrained decoding, so syntax errors become impossible by construction. Then I still validate in code with Pydantic or equivalent, because refusals and edge cases can return outside schema.

Third, I assert on parsed fields, not strings. The fraud test checks parsed label in an allowed set and equals fraud. It never snapshots the surrounding prose. Text eval with fuzzy matching hides drift. Field checks surface it.

Fourth, I pin versions and log fingerprints. Dated model snapshot, not alias. Aliases roll under you and read like model drift. System fingerprint or equivalent on every run, so backend change and near tie flip read as different events.

Fifth, I budget turns and money. A common ceiling I use is around 25 turns before human review. Token budget hard stops runaway loops. K of N voting for decisions that matter. If three runs disagree on a refund, that routes to a person. Disagreement is signal, not a bug to retry away.

Small code, n8n, or Make

All three can host the harness. Pick by who owns it and how often it changes.

Small code fits when a developer owns the path and the steps rarely change. A short script plus schema plus checks, in your repo, under test, under review. In my experience this runs $2k to $6k to build for one strict workflow and the least to operate. It is also the easiest to audit, which matters for money movement and customer data.

n8n fits when the team wants visual orchestration with real agent depth. It integrates with LangChain style tooling, supports vector lookups and multi step reasoning, and self hosts as a free Community Edition from GitHub. Vendor listed Starter is $20 per month for 2,500 executions with annual billing saving about 17 percent. Tradeoff is developer time. Comparisons consistently say n8n needs builders who can debug branches, logs, and state at scale.

Make fits when an operator owns simple automations and speed matters more than agent depth. AI there reads more like an add on than a core runtime. Fine for prototypes and low volume flows. Harder for agent orchestration, memory, feedback loops, and dynamic branching logs. I could not verify current Make unit pricing from its pricing page during research, so I quote it per project after a quick scope check instead of citing a number here.

My rule is simple. One strict workflow with a technical owner gets code. A set of changing integrations with mixed owners gets n8n. Quick departmental glue with low volume gets Make. Anything that touches money, health data, or production deploys gets code plus schema plus human approval no matter which runner hosts it. For background on cost behavior at scale, see orchestration and spend.

What I would do this week

Pick the one workflow that already burned you. The ticket router, the report builder, the intake parser. Freeze the output to an enum plus evidence quotes. Pin the model snapshot. Add schema validation, field assertions, turn budget, and cost budget. Run the eval set N times and record flip rate per item. Treat flippers as low confidence and route them to review.

Most teams land the fix in days, not months. In my experience a single workflow harden runs $3k to $9k depending on tooling and review surface. The payoff is not cleverness. It is boredom. Same input, same shape, same checks, every time. That boredom is what lets a pilot become a workflow.

About the Author

I'm Brian Marvin, an AI-native Fractional CTO with 30 years in technical leadership. At empowered.guru, I help startups build MVPs, shape roadmaps, and make AI-powered technology decisions that scale.