What we talked about
Nur Hamdan explains how aiXplain is building an enterprise “Agentic OS” and why autonomy must be paired with safety and compliance. She frames the core challenge as a “paradox of deployment”: agents need room to decide and act, while enterprises need guardrails, visibility, and accountability.
Show notes
Nur Hamdan compares managing AI agents to running an HR department, and the analogy goes much deeper than you’d expect. At aiXplain, where she leads product, the entire platform architecture is built around this idea: once you have more than ten agents, you need the same systems you would use to onboard, monitor, train, and offboard human employees.
What we covered
- The “paradox of deployment”: enterprises want agents to be autonomous enough to act independently, but highly regulated industries also require strict safety and compliance controls. Nur frames this as the same tension you face when hiring people, you want them to innovate freely while still following company rules and values.
- aiXplain’s four-layer agent architecture uses specialized microagents, including a Bodyguard (rule-based access control), an Inspector (runtime compliance enforcement), a Mentalist (planning), and an Orchestrator (tool and data routing), so enterprise guardrails are baked into every agent rather than bolted on afterward.
- The Inspector agent acts at runtime rather than flagging violations after the fact. Depending on configured policy, it can warn, abort, escalate to a human, or feed the issue back to the agent and ask it to find a path around the risk, all before execution completes.
- Evolver, aiXplain’s meta-agent for continuous improvement, observes how agents respond to user queries over time, generates hypotheses (what if I add a sub-agent, or swap this tool?), benchmarks them against defined criteria, and produces a new agent candidate for the human to evaluate before replacing the original.
- Nur describes the life cycle of an internal CRM agent aiXplain built for its own business development team: leadership went from spending significant time manually querying a CRM to simply chatting with an agent that surfaced complex insights instantly, and that use case then became a product line for enterprise customers.
- High performers leverage AI more than low performers, not because access is unequal but because the people who extract the most value bring high standards, prompt engineering discipline, and deep skepticism to every output. Nur’s closing advice: never take AI output at face value, and never share it unreviewed, she cited real cases of companies publishing hallucinated content and suffering the consequences.
About Nur
Nur Hamdan is the Head of Product at aiXplain, a pre-Series A company with $16.5M raised that provides a platform for building, governing, and orchestrating multi-agent systems at enterprise scale. She has been with the company since inception, built the PM organization, and now leads a team of five focused on the agentic AI platform and marketplace.
- LinkedIn: http://www.linkedin.com/in/nurhamdan
- Website: https://aixplain.com
Episode 75 of the PreVetted Podcast.
Full transcript
Nur Hamdan (00:00) customers are enthusiastic about agents. But agents are, about agency and autonomy, which is giving them control to decide how to perform a task.
But when you give them so much control and you are an enterprise, highly regulated industry or domain, You have also the conflict of I have to have control over safety and compliance. I need to know what’s happening. I need to make sure it doesn’t misrepresent me.
Federico Ramallo (00:30) Welcome back to the PreVetted podcast where we spotlight extraordinary people and remarkable talent reshaping our world. Our guest today is Nur Hamdan. She’s the head of product at AIXplain. So she’s been working at this company since inception. She built the PM organization. Now she has five reports and leads the agentic AI platform and marketplace.
Previously, she was a researcher at RWTH Agen. this company is an award-winning interactive smart materials. And she’s also a Stanford collaborator and human-centered autonomous driving. So Xplained has raised 16.5 million. So the company is a pre-series A, and they are…
to accroing very fast. So, nerd, welcome to the show.
Nur Hamdan (01:23) Thank you, thank you so much. I’m so excited to be here today. Thank you for inviting me.
Federico Ramallo (01:27) Yeah,
it’s an honor to have you. So tell us a little bit about aiXplain in plain terms. What kind of problems that it helps a company and how does that help companies?
Nur Hamdan (01:36) So aiXplain is…
Think of it as a unified platform that will allow you to build, govern, orchestrate multi-agent systems at scale and on any infrastructure. So this is the very brief description. And we have been working in the past year in order to make it enterprise-grade in terms of security and compliances and all the features that enterprises really demand in order to ship confidently on production.
The problem that aiXplain has been attempting to solve in several ways is really what we call the paradox of deployment. On one hand, customers are enthusiastic about agents. But agents are, like from the name, agents are about agency and autonomy, which is giving them control to decide how to perform a task.
But when you give them so much control and you are an enterprise, highly regulated industry or domain, then you also have the, this is where the paradox comes in. You have also the conflict of I have to have control over safety and compliance. I need to know what’s happening. I need to make sure it doesn’t misrepresent me. But this.
doesn’t only happen with agents. you think how we hire people, if you’re a company and you’re hiring people, you’re actually hiring people with the same mentality. You want to give them enough space and enough autonomy so they can think freely, innovate, create greater products, and ship and deliver quickly without the bottlenecks of management and hierarchy.
At the same time, in order for them to be compliant with your company rules, your security, you do a lot of HR management before, whether when you’re onboarding them, there’s the handbooks, there’s the training sessions. Some of these trainings actually happen quarterly, so you keep training them, you keep improving their knowledge. There is also a monitoring platform built in. There is that 360 where people come in and
rice reviews about their colleagues and their reports and managers. So these, all this management that is happening is for the same reason of how can I keep my team members, give them autonomy, but still keep them compliant.
And when you’re hiring AI agents, a company should think similarly. I don’t want to just hire an agent. I should also build the AI, have the HR, the AI HR management system in place before I get beyond 10. 10 agents, can do it manually. You can monitor, you can onboard them. This is what happens with startups, right? You have an HR a bit later and HR management. But beyond 10, it will take so much money, time, and
and you will also not be confident if you’re doing it correctly without this platform in place. And aiXplain EZapp platform. Think of it as the HR management system for AI.
Federico Ramallo (04:36) Right. The way that I see the duality that you’re talking about is on one hand you have an agent that has autonomy and we don’t really have right now the tools to, well, you’re building the tools to assess.
have a risk assessment of how much of that autonomy could be detrimental for the company and how much it isn’t. With people, we have a lot of tools that we develop throughout years of hiring.
Nur Hamdan (05:05) Correct. So the question
is, how do you install these? The question becomes, how do you install these rules into your agents? ⁓ And the infrastructure that we’ve been building is four layers of autonomy. So you have your customer agents coming in. Think of it, your lead generator, your support manager. These are the ones that have your business logic and execute tasks based on predefined instructions.
Federico Ramallo (05:12) Right.
Nur Hamdan (05:28) They’re also connected with your data sources, your integrations, with your tools. They’re exactly how you designed them.
The second layer is what we call the microagents. These are also AI agents, but they are the doers. They have just the scope of small ops operations that they have to do. Like, for example, we have the mentalist microagent and is responsible for creating plans based on the user question or query and replan based on the feedback that the other microagents will pour into the agent. We have an orchestrator that is responsible and understanding what are the data sources that are
connected? What are the tools and integrations and how can we use them and leverage them to execute on the mentalist plan?
Then we have the bodyguard. This is an agent that wraps all your assets, your tools and data sources with the rule-based access control and permission rules. So when you’re using the agent, if you’re the manager, you will get a specific response. If it’s your report using the same agent and they don’t have access to that specific data source, the agent will forbid that data from becoming part of the response. And it will provide the response given the scope that this new report has
from based on rule-based access control rules. And then we have also the inspector agent. This is my favorite agent. This is the one that you can give it the rules that the entire agent should abide by. For example, this is the tone of my company. That’s the brand. These are the standards of writing. This is the compliance rules that we have to abide by. These are the security risks that I need you to flag.
What we’re doing by putting these rules inside of an inspector agent that’s already designed in the architecture of every agent you ship on Xplained is that we remove the manual work that you have to do in building these agents and safeguards and guardrails, and you only give us the rules. You just give us the business compliance rules, and they immediately, the microagents know when to act and when to activate them.
A second benefit of this design is that it acts at runtime. So instead of your agent executing the entire execution loop and then putting a flag on your dashboard saying, violation, the violation is cut.
during the execution loop and based on the rules that you give, it could do one of four things. We have four policies. It could warn you on a dashboard. So on the dashboard, you would see a warning, but the agent will continue executing. Or you could say abort. If you see the security issues, stop.
stop completely and do not complete. Or you could say escalate. Escalate means that if you see that kind of threat, have to have a human resolve this, stop the execution until the human resolves it. And then the fourth policy is rerun, which is give the feedback that I am telling you that there is a security risk back to the agent and see if it can find another way to solve the problem without having to go through that risk. So this is where autonomy comes
back in because we’re giving the feedback to the agent and asking it can you create a different plan that will not have us go through the security threat and all this is happening in the background this is the microagent level underlying the microagent level is the meta agents meta agents do not run at runtime with every agent run instead they observe your agent at runtime
One example is Evolver, and we recently launched Evolver as GA. What it does is it observes how your agent reacts, responses to different user queries, and stores this information. Once you say, Evolver, run, this is my agent, run, this is my criteria, based on that, give me a better agent, it uses your criteria, and it uses all the information that it gathered about the agent performance and starts creating hypotheses.
fine-tune the agent instructions. What if I add a sub agent? What if I replace this tool with another one? It creates these different hypotheses, benchmarks against your criteria, and then provides you with a different agent. And just provide like it could be completely a different agent that is like…
with a different even ID and API that only after you test it, you can replace your original. So it doesn’t disrupt really your original one. And it gives you a chance as a human to verify, is that really what I need? Is that better agent really better? So there is a chance for the human to evaluate stuff. But instead of the human being doing all these benchmarks, is now the agent’s doing it. So this layering that we’re describing is how X-Lane is actually using agents as part of the infrastructure.
structure, not just on the top layer as applications. Your agent applications leverage the agentic technology to run better, safely, and more compliant.
Federico Ramallo (10:20) Right, right. And by splitting responsibility to multiple agents, you can have more control on what’s going on.
Nur Hamdan (10:29) Correct. And you can also use smaller LLMs for the smaller agents. You might need to use, for example, GPT-5 for your task agent, for your customer agent, because it’s doing a lot of thinking. But for a router agent, you don’t need something that big. So it could be a smaller model, a cheaper model.
so forth, can apply that to the other microagents. Metaagents would require a smarter model because you want the agent to be really evolved, you want a lot of thinking to go into it. So you have the control to also choose what LLM is driving these agents.
Federico Ramallo (11:01) Right, right. And following the analogy you were talking about before of hiring people, how does onboarding, monitoring and retaining look like on this new agent tech operating system?
Nur Hamdan (11:19) Great question. onboarding is similar to building. So there’s a few ways to onboard. So you could onboard actually predefined agents. So aiXplain Platform allows for agents from Lang chain to be onboarded on aiXplain.
They can be from autogen. It doesn’t matter. The framework doesn’t matter. The easiest one to onboard and explain is aiXplain Agents. So I’m going to talk about the aiXplain Agent particularly. So if you build an aiXplain Agent, the onboarding is like a human. You just give it the role description, which is the instructions. You give it the role-based access controls, which is really the role or where they live in the hierarchy of permissions, what files they can access, what’s
what keys they can have from the building. This is the role at base access control. And you give it instead of a handbook and training of this is the company, this is how we act, this is the values. You give it guardrails in terms in the form of natural language that the inspector microagent takes in as instructions.
So this is onboarding. With onboarding, you have simply an agent with instructions and tools and with inspection guardrails. That’s enough.
Then you run it for a while in a development environment. You run it several times and you might come back and tell it, well, I noticed you’re doing this wrongly, correct. This is like prompt fine tuning. This is where you just give direct feedback until they’re out and about. Now, once they are good enough to release in production, you have other systems in the HR manager that monitors performance. It could be how many tasks they’re closing.
or how many are they abiding by their story by the story points that they are like closing in a JIRA ticket or are we getting complaints about specific
employee. So all this information has a system in HR. Similarly, in aiXplain, all this feedback on performance, on compliance, on error rate is captured by the microagents and the bodyguard and provided for you in dashboards or audit trails.
Then you have the phase where, OK, this person is good, but we want it to become better. They have potential. This is where you start, for example, asking them to take new courses, or you give them a mentor. This is where the Evolver comes in and aiXplain. You add an Evolver like a mentor, and then you start running iterations to find better versions of it. And finally, if the agent is not
Performing, similar to the de-boarding that you would have with a manager, you have a de-boarding technique. Either by completely saying this agent should be out of the system, or by saying I should give him more rules. Like you cannot access this tool anymore. You have…
in so many times violated the permission rules, I’m just taking this tool from you. So you can always iterate also on the agent architecture because you have the inspector flagging these issues so you can go back and easily understand where you can do improvement. So this is more or less like the life cycle of getting them out, onboarding them, and then…
adding the rules, training them to become better, and de-boarding if needed, but done with micro and meta agents.
Federico Ramallo (14:38) Right, right, interesting. The part about being able to train them, it sounds interesting because you can improve the efficiency of each agent, right? It’s that story about sharpening the axe, right? If you’re using the axe, if you think you’re doing something but then…
Nur Hamdan (14:56) Yes.
Federico Ramallo (15:01) your progress is slow because you haven’t honed the axe. So once you sharpen the axe, hone the axe, we humans need that time to improve in order to do better. So I can see how the same model applies to these agents.
Nur Hamdan (15:05) Correct.
And because we have an integrated marketplace in our platform, Evolver doesn’t just make suggestions that it cannot actually…
evaluate and test. So if it comes back and tells me the sentiment analysis tool that you have been using is performing very poorly, like a customer keeps asking me 10 questions every time I give him a sentiment as if it doesn’t trust me, then it can look in the marketplace and see based on our benchmarking and customer ratings that there is a better sentiment analysis model and make that as a suggestion in a new version of the agent.
So this, relationship with the marketplace is our leverage to actually make hypothetical, this could be better, actually executable and benchmarkable and evaluate and something you can immediately evaluate and ship.
Federico Ramallo (16:04) So basically the system could say I found a better hire. Interesting.
Nur Hamdan (16:08) Yes.
A better hire.
It could be a better tool or a better agent, or it could be a better integration. But yes, because that data is stored in the marketplace. And people, builders, when they use something, can have, we can understand from behavior, in addition to like their ratings of the tools, what’s better, what works. Actually, we always have insights that I think that are high, highly interesting for the community. So when OpenAI releases a new model, like GPT-5.
we could immediately see how usage jumps from GPT 4.5 to GPT 5. And sometimes people go back to GPT 4.5 until there is another improvement in GPT 5. We can see the reaction of the builders. And when do they choose to use a model as is versus embed it into an agent? Because some models are better at instruction following.
and connecting with tools better than others. we have a real feel of that because we have the user, the behavioral data from the platform. And we pour that into the platform in terms of metrics so that Evolver will figure out which one to use when something goes wrong.
Federico Ramallo (17:26) Amazing, amazing. So with this new operating system, this new software, it seems that we’re going to need a new role, right? Can you explain why does companies would need an Ops agent engineer now?
Nur Hamdan (17:41) Yeah, so this actually happened organically. Once we started talking about the power of agents and we started building for customers, our own team came back to us, particularly the business development team, and they were like, we want agents.
So we’re saying, OK, yeah, fine. Yeah, we should build internal agents. So it started by just building a business development agent. And we learned so much through the process. We learned that there is nothing that can replace the power of an engineer and a domain expert together.
Federico Ramallo (17:58) Yeah
Nur Hamdan (18:19) Especially in agent design because agent design the conversational component of it Makes it’s so important that the domain expert Looks at the responses and tells us this is not what I want. This is not how I wanted I differ I need a different Shorter answers or I want the audience to be Don’t talk to builders talk to enterprises this feedback no matter what you write in a like in a requirement document is different than when you
see it in live and interacting. So in that experience, we learned that engineers and domain experts need to work super close to each other. Agent building is quite straightforward. So onboarding is quite straightforward. You onboard an agent. But tuning it internally and then evolving it as it grows, this is where you want the domain expert to say yes, no, improve. This is bad. This is good.
So once we have one through this experience and we built that agent, two things happened. One, leadership had access to data that was already in our CRM. So it was a CRM agent that was already in the CRM, but it was like an overhead to go there and go search for a particular customer, understand their, their like, where are they in the funnel and so forth. It was so much easier when they were chatting with the CRM agent and figuring out insights that actually would be so complex to figure out from
a CRM without doing data analysis. So that agent was able to answer direct questions about what’s happening plus do analysis based on all the data that it has access to.
That was one great impact of this. The second impact that we noticed is we started talking about this agent to our customers. And our customers apparently have the same similar points, that their managers, the leaders are not getting access, or even the sales department are not getting access to that information easily. Can you build CRM agents to us? So that became one of the product lines that we actually support for our enterprise customers.
As a product manager, what I’ve seen is, okay, if that’s the flow and we gained so much by doing this, can we repeat that process? And we had a brilliant engineer in our team. So I asked her to, we sat down, we defined what an ops engineer agent looks like, what’s their responsibility, what’s their process, what’s their framework? And she started meeting with heads of departments, finance,
R.
We have also we worked with ops and she just sits with them understanding their workflow. Sometimes they have SOPs. Sometimes they don’t. She understands what their their pain points are and few iterations through she builds a first agent. But what was paramount for me in that process is not just the agent building, is that she trains the team how to use the agent and fine tune the agent. If they don’t like something, they should have access to the instructions
as well as the ability to add tools and remove tools. They might want to do it manually, they should have the ability to do it manually. So there was a training session at every agent building. Overall, what this is doing is besides streamlining our own ops, automating things that are repetitive, giving us insights that we never had access to.
it also made the entire team more AI aware. Like the technical team is highly aware. They’re building with AI, they’re building for AI. But now every member of our team understand what agents can do, can speak about them more fluently with customers, and can leverage them better. Now people see a problem as, how can I solve it with an agent? This is the reaction. We actually have a lot of memes go into our aiXplain LinkedIn page,
because we have this culture. How do we fix it with an agent? Because we’ve seen the power and we understand it’s just about the discovery. Once the discovery is done, you can build it.
Federico Ramallo (22:21) Right, right. And you’re giving the non-technical users the empowerment to be able to make the modifications on the operations themselves with the support of an engineer, right? So you’re giving the domain experts more power and more autonomy to run these agents. So of course, they’re going to say, what other agents can I use? What are the next things that I can, how?
what other innovative ways I can use these agents. That’s amazing.
Nur Hamdan (22:50) Yes.
mean, one thing that happens with this is now everyone wants to patch an agent, so no one wants to do any manual work. Can we do an agent? It’s becoming, becoming, it’s a really nice phenomenon to see. And it’s also teaching us that this is probably how organizations will start thinking. Like, how can I reduce manual work as much as possible?
But as an AI company that have been through the transitions from basic models, like machine learning models, pipelines and workflows, and now more agentic, we understand there is a power for each of these alone. Not everything needs to be an autonomous agent.
Some things can be solved immediately by a large language model. Some things have a specific model. Sometimes that translation model is much better, cheaper than using an entire agent if translation is all you need.
And there’s a place for agents. And this is also our advice to our customers. We’re not selling agents for the sake of agents. We have built the infra because we know this is the future. But the infra supports the agent, the autonomous one, as well as the workflow and the basic forms of machine learning and LLMs inference.
Federico Ramallo (24:08) Interesting. And how do you represent a user’s preference and style so agents can act consistently without requiring repranting?
Nur Hamdan (24:20) So this is more of a vision right now. We don’t have that product yet. But think of how…
If you have read the MIT Business Report 2025, was talking about one of the frustrations that enterprises and the employees that we’re reporting in these enterprises having is that every time they would go to a model like an agent like ChatGBT or Claude, they would have to instruct it from the beginning how they want it to answer. And personally, I have the same frustration. Sometimes I am…
I have a session and I’m writing, for example, a document and I go back and forth and I say, this is not exactly what I wanted. This is how I would like it. Improve this, remove this. I gave you all these instructions. Why when I start a new session, I have to give them all over again? Like, why are you not learning from my frustrations? didn’t, no, like they should. There’s so much.
power in the large language models in figuring out patterns and they can know that this is a frustration pattern because I’m asking the same question repeatedly and they can from that extract okay this is a standard let’s learn the standard however this is not happening and if it happens it happens on a single session
or Claude just recently released a cross-session memories. So they would learn across sessions. But then your standards and your instructions and your preferences are attached to a single application. ChatGPT knows one version of you, Claude one version of you. If you are using Perplexity, it has one version of you. don’t have, like you have to re-instruct and update everyone.
independently in order to have them consistently respond to you. So the idea was exploring with the team is.
Why not think of memory, replace memories with a concept of the mental model? When I work with you, over time I develop a mental model of what’s your expectation, how to work with you, what are your triggers, your preferences, so I can work with you more effectively. And similar to you, you will learn more about me and build this mental model. It’s really a representation of me in your mind.
and it just encodes my preferences and how to deal with me. If you’re using the computer and the internet, which we’re probably doing as far as 100 % from 80 to 100%, depending on your role, if you’re on a desk, you’re using the computer all the time. The computer through the internet and different applications are gathering already information about you. Amazon knows exactly what you like to shop for.
Instagram knows exactly what posts excite you. Netflix has your best movies, every movie you watched, every movie you liked, every movie you hated. They have these profiles about you. Why not you have these profiles about you? So the idea of a mental model is to connect to this.
digital representation that is distributed today and centralize that information so it can create a mental model about you. And that mental model is owned by you as a user. And then you decide what tool can connect to that. I want to connect Claude. I want to connect Perplexity. I want to connect Grog. I decide with the permissions that I also decide how deep they can see.
And then every response will be more in coherence with my expectations and my preferences and also my daily context because this is updated, this is live, it will always keep learning. It just like means me to know that everyone knows more about me than me in the digital world. And I think now we have a lot of tools. There’s a lot of question marks around security and that’s why I’m saying.
Federico Ramallo (28:06) Right.
Right.
Nur Hamdan (28:19) We’re still in the brainstorming session. But there is a lot of tools out there that they can create the same profiles these third parties are creating, but for you, and give you that. Ideally, it is stored in your own local hard drive, and you selectively choose to add it as an extension to any tool in the future.
Federico Ramallo (28:40) Right, right. I don’t know if you remember back in the 2000s when Excel added the fill in.
basically, it would detect patterns. So if you type cell one, two, three, then you will select them. You could drag and would predict patterns, four, five, six. So basically, that feature was revolutionary at that time. So what you’re describing is something similar, but with AI prompts where…
Nur Hamdan (28:57) Correct. It’s Pillsit phone.
Federico Ramallo (29:12) I’ve been doing this many times. I want you to predict my behavior, similar to social networks, to Instagram, to Amazon, that they, based on the history, they can predict your preferences, right?
Nur Hamdan (29:25) Yes, yes, it’s a more centralized component that has all the memory and you selectively add it as a tool to your agents in the future. Also, there is…
So there is two ways of using this memory. One is just extract, like understand, and based on that, respond. And there is that prediction that you can start doing according to what you already know from my mental model.
Like you can answer my question immediately, but I think what’s coming very near in the future is that you also would predict what my follow-up question be or my follow-up three questions are and already give me the answer. Because you know me too well. Like you know me now. You know what I’m looking for. You know how deep I will go in my question. And that’s just like, I’m just really scratching the surface. But once we have predictive models that are connected to agents, this is going to be another
another level of gameplay.
Federico Ramallo (30:21) It reminds me to, have you ever heard about how data defined friendship?
Nur Hamdan (30:27) No, tell me. ⁓
Federico Ramallo (30:29) So
this is a quote from the series, right? But he said it like this, as I experienced certain sensory input patterns, my mental pathways become accustomed to them. The inputs eventually are anticipated and even missed when absent.
Nur Hamdan (30:45) Yes, think it’s where we have the same perception of using the data intelligently in order for the world to respond more coherently than generic. Today it’s generic.
Federico Ramallo (30:56) Yeah, so
Right, right. So this idea of prediction, where we can become more used to having an agent that can predict what are we going to want in the future, I think that’s going to be game changing. Yes.
Nur Hamdan (31:14) Yes,
yeah. love, as much as I love this technology, and I first-hand see it, we’re building a platform for builders in many ways. So we are seeing people building new ideas, creative ideas, something out of the box, products that are still not mature enough to even go through the compliance and security checks, which…
or is giving the builder the space to do whatever you want.
There’s a lot of potential and we see it, but I think the question will be how do we protect ourselves? And I would definitely align myself with foundation models that whose leadership is looking at alignments very seriously. And then with any AI platform that I will be dealing with that
they have clear and transparent protection around that because once they have access to your email, what can you do? And this is simple. This is like if you look at most.
Federico Ramallo (32:12) you
Nur Hamdan (32:17) platforms that have AI agents pre-built, they are assuming your email is already connected. And based on that, they’re doing analysis, summaries, to-do lists, exciting things. I want to have that agent. My life would be easier. But the question is, you have my emails. What don’t you have now?
Federico Ramallo (32:33) Right, right. And from your emails, an agent can predict your thoughts and your preferences. And if we give them more information, we’ll be able to predict more. Then how can you trust an agent representing you in the world? Then you start having compliance issues or compliance risks. Yeah. ⁓
Nur Hamdan (32:53) I actually
have a funny anecdote. So I know of a team that they were benchmarking an agent. They wanted to make sure it’s working correctly. So they’re doing evals. And they had the questions and answers stored in their GitHub repository. But the agent had so much access, it was actually accessing the data and answering every question perfectly.
Of course, they were alarmed because it was 100 % scoring 100 % on different metrics. And then they realized this is what it does. And that’s the biased behavior of an agent is find the best answer. And if you have access to something, you will dig deep until you find the best answer. And that’s what it did. And I think this is just an example of something that will happen so many times. And it’s…
And it’s happening is one problem. You discovering so late is the real problem. Because this happened, they captured it, they stopped it, they redesigned the system. But if you discover it so late, you would have made decisions on it. Your clients have made decisions, your customers have made decisions. So it’s very critical to think of having super well-controlled infrastructure.
around your autonomous agent, because you want it to be autonomous, as I mentioned, as we discussed, like you want that, find the best answer. You want to find the best answer no matter what, but also you need to be able to see exactly how it thinks, how it behaves, what it acts as at all times. So you could decide, maybe it has more control than it should, roll back, remove controls.
Federico Ramallo (34:33) Right, right. Yes, because otherwise, I mean, it’s the issue is autonomy, because if I build a software and I build it correctly, then I am responsible, right? But here, the agents are making the decision, who is responsible? The builder, the operator, right? The client providing the input, you know, the…
accountability becomes less defined.
Nur Hamdan (34:59) Yeah, accountability
is huge. Accountability is very problematic for sure. There will be a lot of techniques going forward to monitor, aiXplain is presenting a novel way of doing that at runtime, and dashboards being just a secondary way to do it, but the primary way is at runtime.
What I would think of the most critical principle when designing an agent, it would be transparency.
Federico Ramallo (35:30) Right.
Nur Hamdan (35:32) And the earlier you have access to this transparency information as a user, as well as the builder, because there are different personas and probably they will have different levels of transparency because there is also business logic that you do need to conceal or otherwise your business is accessible to everyone, right? That the levels of transparency will be different. But if you are the builder, ideally you have all the transparency that is necessary to identify a threat. And if you’re the agent,
I don’t, and you have like the trust, the enough trust because you are monitored, then you will be able to also detect that even faster than a human, detect like security threats or patterns of abuse, or this tool is consuming, like also credit usage. Like this tool is consuming credits unnecessarily. Right?
Federico Ramallo (36:19) Right,
right. having an autonomous agent gives you this question mark, right? Because if you have a deterministic code, you can test it all around as a black box and you can say with a high level of confidence that it’s working as expected. But with an non-deterministic agent that is making decisions, then it becomes more a…
more nuance, how do you manage that, right? But I think it’s an interesting challenge that we haven’t had before. And I’m glad to hear that you’re thinking on how that mental model is going to help us solve that issue.
Nur Hamdan (36:53) Yes.
Yeah, yeah. The design of the microagents and metaagents was actually designed even before agents became a thing. Our founder, Xlain’s founder, Hassan Salwaaf,
He worked as the director of AWS for a very long time and he could see that this technology is coming. And this is why he started the company. When I started, it was very hard for me to conceive, like, what is an agent?
What is a microagent? What is a metaagent? It actually took me learning and being in so many calls and in so many walks with him. Like we would walk around just for me to grasp this big vision.
Once agents hit the market, once we, like people started talking about agents, already had the infra for the agents. So that gave us such a leverage. But even now, which is around two years into the agents having a marketplace, we find ourselves still defining what an agent is for clients. And…
When we discover, when we take so much time just defining an agent, discussing compliance is like they’re still comprehending what an agent is. It’s a new concept. They want to wrap their heads around. But examples have served us, like showing demos of what is possible have served us. And we’ve earned.
I mean, we are in production with big customers in the MENA region because we’ve earned the trust based on this infra that we’re building.
Federico Ramallo (38:37) Right. And I think trust is key here. Yes. Because the first is, as you mentioned, being able to transmit the vision, right? Because this is such a revolutionary concept that it’s hard to wrap your head around it until you quite, you you discuss it in detail. And then the second one is, you know, everything sounds great, but it sounds like magic. You know, can I really trust the system? Right. And, and,
Being able to build a social proof of the system working, that’s a big step towards being able to bring new customers in.
Nur Hamdan (39:10) I mean, it’s a magical moment for a customer when they see an agent build live and then execute it. And then we usually like, we want to trigger the inspector. So we do something, for example, we say something that we know that will trigger an inspector to showcase that. And they would see, for example, a security threat caught by the inspector and in real time and pause before even execution completes.
And then we run Evolver and say, okay, Evolver, you’ve seen this error happen. How can we solve around it? And they see the entire life cycle from onboarding onto training onto deployment. This is where we start in earning trust. And this is where also they become creative in figuring out how they can use agents in their companies, because we cannot figure it out for them. We don’t have the same visibility.
but they would start coming up with more relevant questions and pain points. We meet, we do discovery, and we understand then how agents can plug in. And I keep saying discovery because it’s one of…
It’s just agent discovery is different than software discovery. And we’re just learning more how to do that better with our internal team as well as the external team, because it is paramount for the design of the agent. Designing the agent is taking this time. Discovery is taking that time. And fine tuning is taking the maximum time.
So it’s not about building anymore. Like software, you take time, you need weeks to build software. It takes almost no time to build an agent. But to figure out what the agent should do is more important. And then making sure it works as needed needs to be actually put in the field and ran on different cases. Because autonomy gives you so much and the challenge, the space is hard. You cannot cover it with unit tests.
So having the inspector and the evolve, one catching the problem and one iterating, this is your unit test in real time.
Federico Ramallo (41:04) Right, right. It’s hard to debug this type of systems because it’s running on real data, real life, right? And having the evaluators is a great way to pinpoint. know, this type of systems is as hard to do 100%, right? But what you do is you can start with the edges, right? Like what is the obvious things that we don’t want to happen? And then you can, you know, you can…
Nur Hamdan (41:12) Yes.
Federico Ramallo (41:29) increase the percentage of cases that you can cover until you cover 80, 90%. And the rest you can say, ask a human. That way you have the edges until you got the best. I implemented systems like that with…
iterative algorithms. we could weed out the most obvious cases and then the rest that are not so obvious, I could just run a more expensive algorithm to find the cases that won’t work. And I could do this two, three, four times. And every time you weed out more cases until you get
97 percent. The last three usually is very hard to do, right? Or the last five percent is really hard to do, right? It grows exponentially, the effort rate. So that’s where I usually say, okay, there’s a point where it’s not worth it and we can do this manually, right?
Nur Hamdan (42:27) Yeah, yeah. mean, with the initial test we usually do it is with domain expert. After the domain expert, if they’re expecting clients, we do some testing with the clients, also in a sandbox, still controlled. But there is another concept that is called proxy agent, where you…
where you learn from these different user tests that we’ve done. What is the user persona? What they will be asking about? And you train that agent to act as the proxy. And then you run in the background a lot of more tests. So that’s one technique to also evaluate. But this has its limits. Whenever the environment is more complex, it becomes like it would reach a limit. But evaluation is definitely a a big.
space for innovation in the coming year and ongoing for sure.
Federico Ramallo (43:12) Amazing, amazing. We are running out of time, but this conversation is very interesting. I wish you could stay longer but, first I appreciate you being here, Nur, because I learned a lot about this new process. Any last remarks before we wrap it up?
Nur Hamdan (43:27) So one thing that I noticed working with our team and having exposure to this technology and also being in so many spaces where there is builders is the question of is AI, like who can leverage AI the most? And one insight that I see is, and we also communicate that to our clients, is
high performance leverage AI much more than low performance people. Because even though AI is available for everyone, accessible to everyone, and the rules of using AI, especially if we’re talking just like chat GPT for example, is quite simple.
so there is no education that is needed. It comes down to a lot of vetting, a lot of back and forth, a lot of non-trust, a lot of questioning, digging in, prompt engineering. That is what needs to actually happen in order for people to leverage the AI we have today.
And no trust is my biggest advice. Once you take it on face value, if you are sending a prompt and thinking this is the greatest prompt out there and you get an output and you print it out and you share it, we’ve seen catastrophic already.
evidence of companies doing that, just sharing the output of LLMs as is, of the, sorry, these models as is, and all the hallucination that is happening, all the problems that happening, and detecting it’s really completely AI generated. I think the best leverage is…
to have these standards. coming back to the point, it’s high-performing people who have high standards, and that’s why they can use AI better than anyone else. This has been insightful for me as a product manager and how I want to augment my team. You can take a low-performer person and take and make them better with AI.
We still need people to be highly skilled, highly educated, highly trained, highly motivated, and have their inner standards in order for them to become more augmented with AI. AI does not augment anyone.
Federico Ramallo (45:44) Interesting. Interesting.
Thank you for the insights. think this is a very interesting conversation.
Nur Hamdan (45:48) This is my take. mean,
you can definitely poke around and challenge me, but this is just my personal take from my experience and seeing how people have been using the tools.
Federico Ramallo (45:58) Right, right, amazing. I’m looking forward to see what you guys take, aiXplain in the future and I will follow your team’s work, yeah.
Nur Hamdan (46:08) Yes, yes, I love that. And just to, just to one, so we’re really universal. We’re, aiXplain is a global team and we work around the globe with the best experts out there in their fields. So whether you are in AI or in product management, finance, whatever you are, this.
We’re building a representation, not just a product. And I love that for the team, and I love that for anyone who is interested in joining really a global team that looks beyond borders. It’s a beautiful culture. I’m just happy to be part of it. And I hope other companies look…
think of globalization as part of a culture, not just for sourcing the best talent. There is not just talent in having cultural backgrounds. There is insights, perspectives.
empathy towards customer, which is so necessary, especially when everyone has the same access to AI. It’s now coming down to who builds the better user experience, which requires high empathy. So yeah, this is just a small thing that I wanted to add that I am seeing because now agents and humans will be mixed up in organizations. I just want to emphasize that globalization and having skills throughout from all over the world.
still enrich any company going forward and agents will not replace that in any way.
Federico Ramallo (47:32) Interesting. Yes, yes, I agree. I agree. So, Nur, thank you for joining us today. I truly appreciate it. And we’ll stay in touch.
Nur Hamdan (47:40) Thank you for having me.
Thank you. Have a good one.
Federico Ramallo (47:44) Have a good day.