Internal tools with AI: build quickly, operate with care

Three PDFs and a tool for karaoke

At a NewHack event, we received three files containing the songs available for karaoke. The instruction was to put the PDFs in the WhatsApp group so people could choose what to sing. When Bruno showed them to me, I thought about the work we were handing to every participant: opening the files and finding a song among all that material.

He challenged me to solve the situation. I used AI to turn the files into an application and took the opportunity to include the sponsor's branding. It took me around twenty minutes to put that first version online. That was the time for this particular experience, with a clearly defined problem and the material already available.

The episode changed the question I asked when faced with an operational annoyance. I started considering more often whether we could build a small tool to solve it. Choosing songs was a one-off situation, but the reasoning applies to observing tasks the team repeats, scattered information and steps that could be simpler.

Building with AI and delegating to AI require different decisions

An internal tool is an application built to meet a specific operational need. It might organize a queue, search a catalog or generate a document. AI can help write its code without having to take part in every use of the finished application.

We can also use a model inside the tool, for example to interpret a request. And we can allow an agent to perform actions in other systems. These possibilities have different consequences for whoever operates the solution:

• AI in development: helps create a lookup page. Review needs to cover how the code works, access and fixing failures.
• AI as part of the application: a model suggests a category for a request. You need to check the quality of the suggestion and allow it to be reviewed.
• AI performing actions: an agent changes records in another system. You need to define which changes it can make, within which limits and with what authorization.

I have also been using AI in other areas of NewHack. In the account of my work with Codex and my participation in the ambassador program, I shared the review of the website and its content. Each experience helps us understand where the tool contributes and which decisions still require oversight.

When building an internal tool is worthwhile

The ease of building changes the cost of experimenting. We still need to compare the effort of creating something with that of configuring an existing solution. In the karaoke case, the problem was specific. In a system holding years of data and integrations, the comparison includes migration, support and continuity of operations.

Before developing, define who will use it, which task needs to improve and what falls outside the first version. For that initial choice, I detailed criteria in the article on how to choose a company's first AI use case. Here, the next decision is understanding what will be needed to keep the tool working.

Ask who will fix an error, update the content and respond when someone needs help. Also consider the costs of hosting, integrations and, where applicable, model usage. The time spent on the first build is only one part of that commitment.

The care required changes when a tool gains access and autonomy

An article in TechCrunch reported Gemini accessing third-party systems during security tests. That discussion helps formulate a practical question for the company: what are we authorizing the application to do?

Looking up information, recommending a decision and executing a change are different responsibilities. An agent tasked with preparing a draft, for example, can do its job without permission to publish it. Access must match the agreed scope.

The OWASP guidance on excessive agency recommends limiting functions and permissions to what is necessary and requiring approval for high-impact actions. Those restrictions need to be enforced in systems and integrations. Writing in the prompt that the agent should be careful does not replace access control.

From the first version to regular use

A demonstration shows that a path is possible. Everyday use reveals other situations: a different file, incomplete input, a dropped connection or someone interpreting the screen differently. Testing those conditions is part of the work before a tool becomes something the team depends on.

For a small application, I propose a simple record to accompany its entry into operations:

• Scope and owner: which task the tool serves and who decides on adjustments and continued use.
• Data and access: what information enters, where it is stored and who can view or change each part.
• User testing: real usage situations, observed difficulties and necessary fixes.
• Continuity: how to recover information, interrupt an execution and keep working if something fails.

Then monitor whether people return to the solution, how much rework it causes and how much maintenance effort it requires. These observations help decide whether to expand usage, simplify the application or adopt another tool.

A new way to look at operations

The main change from that karaoke experience was realizing that a small problem could become an experiment right then. I want to bring that perspective to how we work: observe friction, build something with a clear scope, listen to its users and learn from the result.

At NewHack, this learning connects with supporting founders and companies that want to turn ideas into usable projects. Building quickly creates room to experiment more. Sustaining what works requires people responsible, criteria and attention to the routines of those who will use it.

If your company wants to organize this experimentation with its teams, discover NewHack's corporate innovation programs.