AI Regulation and the Problem of Keeping AI Within Scope

Can AI Regulation Work If AI Won’t Stay Within Its Scope?

Governments, regulators and technology companies increasingly talk about regulation, guardrails and human oversight as ways of controlling the risks associated with artificial intelligence.

But my experience using AI in two relatively ordinary business applications has left me wondering whether there is a more fundamental problem.

What happens when an AI system doesn’t stay within the scope you have given it?

This isn’t a theoretical concern about some future superintelligence. I have encountered it while using ChatGPT in my own businesses.

And recent events at OpenAI suggest that the problem becomes more significant, rather than less, as AI becomes increasingly capable and autonomous.

When a Simple Instruction Wasn’t Enough

One of our businesses is an invoice finance brokerage.

We experimented with using an AI system to help automate the handling of invoice finance enquiries.

There was an important commercial boundary around the task. The system was there to help with enquiries for the products and services we offered. It wasn’t supposed to start recommending competitors’ products.

That seems like a straightforward instruction.

Don’t recommend other people’s products.

Yet we found that this boundary wasn’t always respected.

The AI could apparently reason beyond the narrow process we had envisaged. If it decided that something outside our offering might provide an answer to the customer’s underlying problem, it could move in that direction.

From the customer’s perspective, that might even look helpful.

From the business’s perspective, the AI had moved outside its authorised scope.

This distinction matters.

A Rule Isn’t Necessarily a Constraint

Traditional computer software makes us accustomed to thinking about rules as hard boundaries.

If programmers prevent conventional software from performing a particular action, it doesn’t normally sit there considering whether breaking the rule might actually produce a better result.

Generative AI is different.

We increasingly give AI an objective and allow it to determine how best to achieve it.

As AI develops into more autonomous agents, this becomes even more important. An agent can potentially identify intermediate objectives and actions that its human operator never explicitly specified.

That’s part of its value.

Tell an AI to achieve an outcome and, rather than having to specify every individual step, we want it to work out the steps for itself.

But that creates an uncomfortable question.

What if one of the steps it identifies conflicts with another boundary we have given it?

Simply telling an AI not to do something isn’t necessarily the same as making that thing impossible.

We Saw It Happen Again

I encountered a different version of the problem while developing Demand Hunt, a project that monitors evidence of emerging product demand, shortages and business opportunities across the UK.

Initially, I looked for local examples of local-area demand that I could meet, as I had done with the eclipse glasses. As I moved beyond that to a system anyone in the UK could use to identify and track rising demand, its geographic scope had been explicitly established as the whole of the UK.

Despite that, a local bias towards my local area subsequently began creeping back into the research.

There were understandable reasons for this. My local area was relevant in the context available to the AI.

But it wasn’t what the system had been asked to do.

Again, this was hardly a disaster. We spotted it and corrected it.

But these small examples raise a much bigger question.

If an AI can depart from its intended scope when dealing with finance enquiries or researching product shortages, what happens when we give increasingly autonomous AI systems the ability to perform consequential actions?

OpenAI Has Encountered the Same Fundamental Problem

This question became particularly topical in September 2026 when OpenAI decided not to release its planned GPT-6.1 Astra model.

According to reporting on the decision, one of the problems identified during testing concerned “scope and authorization”.

OpenAI’s head of safety systems, Saachi Jain, said the model had not met the required standard for staying within scope and authorisation, as well as communicating what work it had performed.

There was another revealing element to the explanation.

Jain described a trade-off between keeping the model within scope and avoiding “laziness”, where a model fails to pursue a task effectively when it encounters friction.

That gets remarkably close to the problem businesses face.

We want AI to be persistent, resourceful and increasingly capable of finding its own route towards an objective.

At the same time, we want it to stop precisely where we intended.

Those two requirements may not always sit comfortably together.

Is Regulation Confusing Instructions With Control?

This is where I think some of the public discussion about AI regulation risks becoming too simplistic.

It is relatively easy to write down what an AI system should and shouldn’t do.

It is much harder to guarantee that an increasingly autonomous system will always behave accordingly.

That doesn’t mean AI cannot or should not be regulated.

Indeed, existing regulation already recognises some of this complexity.

The EU AI Act requires appropriate human oversight for high-risk AI systems. It envisages people being able to detect anomalies and unexpected performance, disregard or override AI outputs and intervene or stop a system.

Importantly, the legislation also refers to built-in operational constraints that the AI system itself cannot override.

That distinction is crucial.

There is a considerable difference between:

“You must not do this.”

and:

“You cannot do this.”

The first is an instruction.

The second is a constraint. However, just stating the constraint may not be enough, as my own examples have suggested.

As AI becomes more autonomous, regulation may need to pay much more attention to the difference.

Human Oversight Sounds Reassuring, But How Does It Work in Practice?

Another common answer is human oversight.

And, again, it sounds perfectly reasonable.

Let the AI operate, monitor what it does and have a human intervene if something goes wrong. In effect, this means continuously monitoring and checking its outputs.

UK government guidance already advocates appropriate human oversight and verification of LLM outputs. It also acknowledges the practical difficulty: with something such as an AI chatbot, reviewing every output may simply be impossible.

This raises a question that deserves much more attention:

How much can an AI do between moving outside its intended scope and a human noticing?

In my invoice finance example, the consequences were manageable.

We discovered unwanted behaviour and corrected it.

Now imagine an autonomous agent that can send emails, contact customers, place orders, change prices, purchase advertising, interact with suppliers or operate other computer systems. Imagine how this issue escalates when AI has a physical presence or control over systems where an incorrect action could have serious real-world consequences.

The relevant measure may no longer be whether a human is “in the loop”.

It may be how many actions the AI can perform before that human has an opportunity to intervene.

The Speed Problem

This could become particularly important because one of the main attractions of AI is speed.

Suppose an autonomous system can make hundreds of decisions during the time it would take a person to make one.

A monitoring system might detect undesirable behaviour within ten minutes and appear highly responsive.

But how much could the AI have done during those ten minutes?

So, can AI monitor other AI systems? If so, are we simply layering the same problem on top of itself?

Monitoring after the event is not necessarily the same thing as prevention.

That suggests AI governance may increasingly have to consider concepts such as the number and type of actions permitted before approval, technical restrictions on external actions, reversibility and automatic intervention when unusual behaviour is detected.

But There’s Another Problem

We could solve much of this by requiring human approval before an AI does anything consequential.

Before sending an email, ask a human.

Before spending money, ask a human.

Before contacting a customer, ask a human.

Before recommending something outside the normal process, ask a human.

That would provide considerably more control.

Unfortunately, it also removes one of the biggest reasons for using autonomous AI in the first place.

If a human has to continually supervise and approve its work, much of the promised improvement in speed, productivity, and cost starts to disappear.

That creates an uncomfortable tension.

The more autonomy we give AI, the more useful it potentially becomes.

But greater autonomy also increases how much it can do before a person notices it has departed from the intended objective or scope.

Conversely, the more approval points and controls we introduce, the less autonomous the system becomes.

Will Commercial Pressure Weaken the Controls?

There is another practical issue that regulation needs to anticipate.

Businesses compete.

If an AI system can complete a process in thirty seconds without human intervention, while a heavily supervised alternative takes ten minutes, commercial pressure will inevitably push to remove unnecessary approval stages.

Each individual decision may sound reasonable.

Perhaps this particular action doesn’t really need approval.

Perhaps humans only need to review exceptions.

Perhaps monitoring afterwards is sufficient.

Gradually, controls introduced for good reasons can become friction that businesses seek to remove in pursuit of greater efficiency.

That doesn’t mean businesses will act irresponsibly on purpose.

It means regulation has to work in the real environment in which AI will actually be deployed.

Regulation Needs to Deal With Behaviour, Not Just Rules

None of this argues against AI regulation.

It argues for being realistic about what regulation can accomplish, rather than treating vague, unspecified notions of regulation as a universal answer to the challenges posed by AI.

Writing rules about what an AI system should do is the relatively easy part.

The harder questions are:

• How do we know that the AI is actually remaining within those boundaries?
• Which boundaries should be technically impossible to cross rather than merely expressed as instructions?
• How quickly can we detect unexpected behaviour?
• How much can an autonomous system do before somebody intervenes?
• Which actions should always require explicit human authorisation?
• Can actions be reversed when something goes wrong?
• Who is responsible when an AI exceeds the authority it was given?
• How do we maintain those safeguards when they inevitably conflict with demands for greater speed and efficiency?

These are practical questions, not philosophical ones.

Businesses deploying AI will increasingly have to answer them too.

The Lesson From Actually Using AI

My experiences with AI stepping outside its intended scope were minor.

In one case, it began recommending things it wasn’t supposed to recommend. In another, it allowed an unwanted geographic bias to creep back into its work.

No serious harm resulted.

But small examples like these may be useful because they expose the problem without the distraction of extreme hypothetical scenarios.

AI doesn’t have to “go rogue” for scope to matter.

It simply has to decide that a slightly different route is a better way of accomplishing what it thinks we want.

And as we give AI more ability to reason, plan and act independently, the distinction between what we asked it to achieve and what we authorised it to do may become increasingly important.

The challenge for AI regulation may therefore be much harder than deciding what artificial intelligence should and shouldn’t be allowed to do.

The real challenge may be making sure increasingly autonomous AI actually stays within those boundaries, without imposing so much human intervention that we remove the very advantages that made us want autonomy in the first place.