AI in Production

The Demo Is the Easy Part: Lessons from the First PLATFORMday San Francisco

On November 6, platform engineers gathered at Sentry's San Francisco office for the city's first PLATFORMday. Four sessions, including mine, kept landing on the same point: building with AI is no longer the hard part, and proving that it works is.

By Gabe Gramajo  ·  Founder & CEO, Ideas Factory  ·  November 8, 2026

I went to PLATFORMday San Francisco to give a talk called "Beyond the AI Demo," built from what we see deploying AI for small and midsize businesses. Most of the room were platform engineers, people who run infrastructure for much larger teams than any of my clients will ever have.

What struck me was how often the other sessions arrived at my argument from completely different directions. Here is what I took away, and why it matters even if your business will never have a platform team.

Launching AI is easy. Landing it is not.

The line of the day came from an enterprise systems leader at a crypto exchange: "Launching AI is very easy, but landing is difficult."

His point was about order. Most teams build the workflow, add agents, and only then ask about return on investment. He argued for the reverse: set the business objective first, decide how you will measure it, and only then build. Six months of building before anyone asks "was this worth it" is how AI budgets get cut.

Two details stuck with me. The first is that almost nobody knows their baseline. When his team asked people how long one task took, they guessed six hours, three hours and two hours. Measured with a tool, it took one hour.

If your "before" number is a guess, your savings number is fiction.

The second is attribution. A bug that used to take six days did not drop to thirty minutes with AI. Once you count the engineer's research, messages and calls, the honest number was about four and a half hours. Still a big win, but a real one.

He also sorted AI work by two questions: how much knowledge does the task need, and what does an error cost? A first contract draft is a "no regret" use. Financial statements are "human first." In his words, "We don't want AI to create any dollar amount in the financial statements." An agent can gather the data for a 30-hour trial balance across about 100 ledgers. A controller still reviews it.

PRs are cheap. Outcomes are not.

The next session looked at AI coding tools from an unusual angle: the session itself. Instead of counting pull requests, the speaker analyzed the full conversations between developers and their AI tools, down to the sentiment along the way. A session that ends with a satisfied engineer is a better sign of an outcome than a merged pull request, because, as he put it, pull requests are cheap.

He was just as blunt about reporting: "the dashboard's where data goes to die." The value comes from acting on the analysis. In one what-if review, a cheaper model would have handled 70 of 74 sessions just as well. Time spent searching files and wrestling with version control is cost with no outcome attached.

His other point is one every business owner should hear. Those session records are your data, not your vendor's, and cloud tools often discard them when a virtual machine shuts down. Keep them on purpose, keep summaries rather than raw prompts, and scrub personal information before anything leaves the machine.

The most dangerous API is the one you forgot

A security-focused session took on deprecated APIs: endpoints that were supposed to be retired but still answer requests in production. Security teams chase active threats. These forgotten endpoints are passive risk, and they usually surface only after something goes wrong.

The distinction I wrote down was enforcement over governance. Telling a developer to delete an old endpoint is governance. An automated system that watches traffic, removes the endpoint, and removes it again if a rollback brings it back is enforcement. If you have no traffic history at all, you start with an inventory: walk the code or scan the system to build a first snapshot.

His closing rule, paraphrased: the most dangerous API in your system is often not the one under attack. It is the one that has been sitting there, forgotten.

Boring on purpose

The organizers also ran a hands-on session on Reliaburger, an open-source, lightweight alternative to Kubernetes that started as a white paper in February 2026. The pitch was refreshingly modest: cover about 80% of what most teams need, leave the rest out on purpose to reduce cognitive load, and keep it simple enough that you do not need a dedicated team to run it. The live install on a laptop took a few minutes.

The line I liked best: "We just want it to be boring and useful." That is the right standard for any system a small business will depend on.

What I brought: beyond the AI demo

My session made the same argument from the small business side, and the market numbers are sobering. S&P Global Market Intelligence found that 42% of companies abandoned most of their AI initiatives in 2025, up from 17% the year before, and that on average 46% of proofs of concept were scrapped before production. Among small businesses, Goldman Sachs found that 76% use AI, but only 14% have it fully integrated into core operations. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027.

The reasons behind those numbers are cost, unclear value and fit with real workflows. The model itself is rarely the problem.

I walked the room through one real order on Vision Concierge, our guest-facing ordering and assistant platform: a guest at a brewery orders boba from the tea shop next door. The AI's job is to understand the request. Everything after that is integration: checking the item and price against the shop's point of sale, writing the order there, and confirming the guest only after the system accepts it.

That is the boundary I care most about. "Your order is in" is a response. An order in the point of sale is a commit. A drink at the guest's table is the outcome. Only the last two prove the automation worked.

I also shared two lessons we paid for. A weekly report for a lodging client failed quietly because one of its four booking sources was never switched on, which taught us that silence is a failure mode. And we lost a pilot because we treated the customer's configuration as their homework instead of our critical path.

The thread that tied the day together

Four sessions, four very different audiences, one shared set of rules:

1. Measure the baseline before you build.

   If you cannot say how long the work takes today, you cannot claim AI saved anything.

2. Count outcomes, not activity.

   Pull requests, conversation counts and token totals show the tools are busy. Completed orders, closed tickets and verified hours saved show the business is better.

3. Turn rules into enforcement.

   A policy someone has to remember will be forgotten. An automated check that alerts on missing success, or removes a dead endpoint, keeps working.

4. Choose boring systems you can operate.

   The right platform for a small team is the one it can run, recover and understand at 9 PM.

What this means for your business

You do not need a platform team to apply any of this. Pick one recurring task that leaks time or revenue. Measure it for two weeks before you change anything. Define what "done" means in your own system, not in the chatbot. Then automate it, and count completed outcomes rather than conversations.