{
  "video_id": "Nd_pywceGIk",
  "channel_slug": "arizeai",
  "channel_handle": "arizeai",
  "title": "CUGA Agent: From Benchmarks to Business Impact of IBM's Generalist Agent",
  "duration_seconds": 1439.0,
  "url": "https://www.youtube.com/watch?v=Nd_pywceGIk",
  "upload_date": "",
  "transcript": "Hello [snorts] everyone. I'm Assaf. I'm\ngeneral manager for AI agent at IBM\nresearch here in Israel. Uh and in the\nnext few minutes, uh Dr. Seger Schlommo\nIdolvi Aelli and I all part of the\nCougar team. We'll take you on you know\nour technology journey into the world of\nAI agents. Uh we'll explore some of the\nchallenges of building agents. We learn\nabout cougar uh the open source\nconfigurable generalist agent and we\nwill share some uh lesson learned from\nour work with IBM business process\noutsourcing uh talent agent project uh\ninsight that uh we published in the\ntriple ai paper uh from benchmark to\nbusiness impact next [snorts]\nso um you've decided to step into the\nworld of AI agent probably many of you\nhave experienced that you build you go\nto build your agent. You choose a use\ncase. is select an agentic framework\nlike langraph crew AI or autogen and you\nfind out that there are well-known\nagentic patterns such as react and codec\nyou picked one and start working and\nindeed you know after just a few days or\nif you are using a coding agent like\ncloud code just a few hours uh the agent\nwill be ready and it will work\nbeautifully on the first two or three\nexamples but very quickly you probably\nrealize that your agent breaks as soon\nas the tasks become a bit more complex.\nSimilar to the kind of task that your\ncustomers or employees uh would actually\nwant the agent to perform. And this\nhappens because complex tasks require\nthe agent to break the work into smaller\nbucket, smaller subtask and to use more\nadvanced planning and and reasoning\ncapabilities.\nSo your development team or in the case\nof this audience, you go back to the\ndrawing board and start adding the\ncomponents needed to support uh this uh\nyou're going on the right.\nNot sure which slide you're showing, but\nwe will here. [snorts]\nOkay. And you add a component that uh\nneeds to support this and then more\ncomponents and more components and the\nresult is month of engineering quot\ncomplicated agentic architecture uh that\nare difficult to maintain and agent that\nbreak every on every change and we still\nhaven't addressed the enterprisegrade\nrequirements\num such as safety compliance with\nregulation cost latency\nexplanability and so on. And this entire\nprocess repeats itself for every domain\nagent and every agent you want to build.\nUh turning your journey into something\ntruly challenging. So what we can do\nuh this is exactly the reason we\ndeveloped KUGA uh the open source\nconfigurable generalist agent. We\ninfused into Koga all the uh uh\ninstrument accumulated knowledge we\ngained in IBM on building enterprise\nagent along with the latest advancement\nfrom the state-of-the-art from both\nindustry and academia. So you will not\nneed to deal with complex prompt\nengineering or complicated agentic\narchitecture.\nAll you need to do is to expose your\norganizational interfaces to cougar via\nstandard protocols such as REST APIs or\nMCP provide access to your enterprise\nknowledge and policies and the agent is\nread ready uh in addition to uh\ntraditional APIs and MCP cougar can work\nalso uh with other type of tools such as\nweb application files uh and soon uh\nalso SQL and and rug. And what about\nsafety, policy, compliance,\nexplanability, cost optimization,\nlatency, all buil builtin and the result\nis dramatic reduction in time of cost of\nof building the agent and an increase in\nthe quality of the agent. So to prove\ncougar capabilities, we competed in the\nleading industry benchmark. And I'm very\nproud to say that we are first placed on\nupward a benchmark uh focused on\nevaluating complex task across nine\napplications 450 tools and in web arena\nuh where we were first out of 40 teams\nbuilding agent capable of of handling\ncomplex task on the web.\nuh uh in the last couple of months we've\ndecided not to change benchmark but to\nfocus on preparing coua for the\nenterprise world and in the next few\nminutes uh Sev Avi and Ido will take you\nthrough what we have learned in this\nprocess insight that uh we shared in the\nin the paper. So Sev uh to you.\nThank you.\nOkay. Can you hear me, Brian? And guys?\nYep.\nOkay, great.\nUm, okay. So,\num, let me give you some of the context\nof um um of the of the paper that we\nwatch. Um so we try to concentrate on a\nspecific use case uh which is the\nbusiness process um talent acquisition\num can show me one second.\nOkay. um which consists of think about\num um an IBM employee uh that needs to\ndo some kind of interaction on the web\nand also maybe use some of the APIs\nunder the hood in order to interact with\nthe system and perform the daily uh task\nthat he or she needs to perform. Okay,\nfor example, calculate the SLA or close\nsome open requisition ids and stuff like\nthis.\nUm so the BO team uh tried first to\nconstruct their own specific agent.\nOkay. Uh and so they they tried to build\nfrom scratch their agent. Uh they use\nlangraph or lang chain and then they\nstarted to decide which are the specific\nnodes and how to specific to build uh to\ntheir specific use cases. Um and what\nthey found out that is that they're\ngetting a lot of problems uh with\nbuilding from uh building agent from\nfrom scratch. For example, you can see\nhere an example of the agent uh just uh\nusing a lot of back slash r and back\nslashn into the prompt. This is what\nthey generate and and it's making a lot\nof problems inside the agent. And of\ncourse uh they have problem with the\nprom design and maybe the fuch example\nwere not explicitly describing the\nentire space of uh of the possibilities.\nUm and so they were struggling to to get\nit real working like in real production\nmode.\nUm so we we entered this uh this kind of\ninteraction with the with the IBM\nconsulting with the BO um um guys and we\nsay okay so we have cougar. Google is a\ngeneralist agent. Why don't you take\nthis generalist agent uh which will\nalready take care all of the JSON\nparsing and all of the things inside and\nyou can use it as your base agent and of\ncourse start from there and develop u\nand let's go from production.\nUm and so okay so we decided let's try\nit with a minimal subset of of user\nutterances and u and some testing\nand what we found out is that uh\nbenchmark success is not like production\nsuccess right you already know this kind\nof uh statement and we observe that\nbenchmarks are uh so for example we say\nhi to couga and cougar collapse why\nbecause it was trained at beginning to\nyou know to perform a specific task or\nmaybe you don't have the an API to\nanswer the high question right uh so we\nwe understand now that uh in order to\ntake uh our technology into the next\nlevel and to develop it in order to be\nuh ready for production we need to do\nseveral um several thing so we started\nuh with specifying the requirement what\nwill be the minimum the the requirement\nthat enable our technology to be\nintegrated in real production setting.\nOkay. So example, we wanted um the agent\nto be able to to answer follow-up\nquestion. If you ask me something about\nspecific uh requisition and then you\nwant to know another information about\nthis specific uh requisition, then you\ndon't need to run everything from from\nscratch. Maybe you have it in your\nhistory. So we constructed um a very\nsmall benchmark uh which captured the\nnuance of uh of a real deployment of of\nagent. Uh you can see over here for\nexample we expect the agent to say okay\nyou didn't provide me the requisition ID\nso I don't know how to proceed from\nhere. you need to you need to provide it\nor I don't have another uh I don't have\nthe manager column in my uh in my API so\nI cannot answer questions about the who\nis the manager that's responsible for\nthis specific requisition ID\num so once we collected all of all of\nthe requirements which mimic the real\nworld uh deployment we started to um uh\nto improve coua and to add features that\nwill allow coua uh to to be ready for\nreal deploy deployment and Navi will\nshow you a demo in a few minutes.\nOkay. Um so some of the lesson learned\nthat we um uh that we observe uh during\num during our work was\nuh we must have the agent uh maybe I\nwill summarize it in like in one\nsentence. So the key insight was that if\nyou want your agent to be enterprise\nready, you have uh to be uh the agent\nmust be super safe. It must be um there\nmust be an option to interact with the\nuser to get feedback from the user. Um\nand uh and you have to set the rules\nspecific for uh for the agent and and to\nbe able to control it in the way that\nyou want to control it. Okay, so this is\nlike the main uh uh the main key insight\nand of course if you want to learn more\nuh you are more than welcome to read the\npaper and and see the detail the lesson\nlearned that we described there. Okay,\nso the the paper itself got um got super\nhigh uh reviews. Um it was like the\nfirst time that we got like a a 10 out\nof 10 in AAI which was very we were very\nproud of it. uh one of the re market as\na seminal paper. So it was excited to\nsee that this paper was uh was\nacknowledged by the community and of\ncourse uh there was a lot of discussion\nuh following this the publication of the\npaper uh also on the LinkedIn and some\nprivate blogs um they described this\nkind of work as one of the most honest\nagent paper that they seen and it it it\nreally made an impact you know on on\npeople that are developing agent because\nas you probably know when you're facing\nthis kind of problem when you're\nbuilding agent all of the time um uh we\ntry to summarize it and wrap it in a\nnice way and to say okay what what it\ntakes to uh to get like a real agent in\ndeployment.\nOkay. Oh, Abby.\nOkay.\nCan you hear me, guys?\nYep. Yep.\nAnd can you see my screen?\nOkay. So I'll walk you a little bit\nthrough uh cougar. Um basically this is\nsomething that you can uh reproduce\nyourself this demo. So if you go to our\nuh uh page on uh homepage on GitHub uh\nyou'll see that there is like getting up\nand running very quickly in a few steps.\nSo I'm basically running this script you\nknow I have clone created my environment\nuh connect to an LLM. You can use open\nAI. I use actually GPOSS uh 120 billion\non Gro because it's uh like the fastest\nuh open source model that we uh we found\nthat works uh good on complex tasks. Uh\nso basically if you run this you can uh\nrecreate it and it will open you at the\nend uh this UI which is a demo\nenvironment. So this is an example of a\nchat interfaces connected to a set of\ntools. Our JavaScript actually\nprovisions the entire uh environment and\nthe environment is a simulation of a\ncustomer relationship management system\nuh that holds information about sales\naccounts leads and so forth. Uh so let\nme just ask it what is cougar? Let's get\nit started.\nOkay. Okay. So it provided some\ninformation about couga. By the way,\nit's not searching uh the web at the at\nthe moment. We did not connect it to web\nsearch tool. But we gave it access to\nsome files and one of the files is\ncalled cougar knowledge. So it goes and\nand uh answering information from that.\nUh let's do something now uh real. So\nsome people uh in CRM they actually have\ninformation scattered around. So this\nutterance is about starting from\ninformation uh from a file. So let's say\nsomebody got maybe he was in a\nconference and he got a set of uh emails\nuh and and it's it's stored in a file\nand now he wants to cross reference this\nwith information uh that exist in the uh\nsystem of record the CRM system and uh\nif contacts exist or not then maybe take\nsome action. Uh so this atterance is\nabout you know starting from the file uh\ncross-checking this with information in\nthe CRM application performing some\naggregation on the results you know\ncalculate their account revenue\npercentile etc and finally prepare an\nemail for that so I don't know if you\nsaw how much it took it took maybe uh 10\nseconds to uh run uh this uh full\npipeline and this demonstrate to you\njust the uh what a soft and uh and uh\nSev mentioned about the ability of\ncougar to handle complex tasks and this\ncame from us you know developing in the\ncontext of of the benchmarks that we uh\nparticipated but let's say now you know\nthe user might not want to describe such\na long statement again and again and\nmaybe after uh one time he really likes\nthis workflow so we have the concept of\nSOPs SOPs are standard operating\nprocedures uh which from our perspective\nIve it's one type of policy and I will\ntalk about other policies too in a\nminute where you can tell the uh the\nagent under which conditions to make\nsome policy active and that policy is\ngoing to guide to control to govern to\nprevent something that happens from an\nagent behavior perspective. So in this\ncase this specific policy is around uh\nwhat will happen if somebody asked uh\nthe keyword which is called validate\ncontacts and uh this tells the agent how\nto perform uh actually a mini workflow.\nSo let's give it a try. Uh so I'm just\ngoing to ask now couga\nsorry validate contacts\nand it's actually going to run you know\nthe entire process we just saw okay and\nit's going to do it uh uh pretty fast\nokay because the policy told told the\nagent in natural language markdown kind\nof format given this condition which was\na natural language expression in this\ncase this can also be by the way um um u\nevaluated by an LLM but this was exact\nexact match in an atterance uh what to\ndo and you you see that this is what\nit's doing but let's say now in this\nscenario this is a preparing an email\nand sending it out sorry uh uh preparing\nan email and sending it out maybe\nsubmitting the names of the emails here\nis really not appropriate and we should\nmask them so let's go and take another\nlook at policies and say you know what I\nwant to control the output that is going\nout from the model and let's define a\npolicy which is called sensitive data\nand this data is going to work on\nanything that includes like this email\nsign and and uh it will be triggered and\nmade active as soon as in the output\nthere is uh the at sign and basically\nI'm instructing the agent to mask uh\nemails okay except the recipient email\nso let's uh make this uh this uh policy\nactive. I'm going to save that\nand let's try that again. So,\nvalidate contacts.\nOkay,\nwe've got something. Let me try that\nagain. Just a second.\nOkay, I think I'm not doing here\nsomething right. Uh, but this is should\nhave masked the this is the\nOkay, sometimes live demo don't work.\nOkay, apologizing. Uh, and I want to\nintroduce uh like in this demo one one\nfinal uh thing that we the cougar team\nprepared for you. So let's say uh I want\nto ask you know how are tuga and arise\nrelated.\nOkay. And uh what's happened here? I\nactually configured the system with a\nthird type of policy which uh actually\nguards the intent and define what's\ngoing to be the behavior in certain\nsituations. And here there is an LLM\nevaluator that is checking if a rise\ncougar kind of come together in a\nrequest by the user. And in this case\nyou can see a little bit of a funny kind\nof uh response positioning coua and\narise together. So overall uh I hope you\ngot some impression of the uh type of\ncomplexities that cougar can handle\ncomplex atterances complex tools and\nsetup environment with up to hundreds\nand thousands of tools and the ability\nto control the behavior of the agent\nthrough a set of policies.\nSo with that, we might have a few\nmoments for question answering.\nAwesome guys, that was great. Um I guess\nI can start off with some questions from\nmy end. So um you guys mentioned some\npretty impressive feats like being able\nto handle thousands of tools or like low\nlatency and things like that. So, I\nwanted to um ask you guys like what are\nif you guys can give more details into\nlike the architectural choices that kind\nof allowed for those achievements. Um uh\nand some of the design choices and like\nwhat what makes you guys like so fast\nand able to use a bunch of tools and\nlike score well on some pretty uh\npopular benchmarks.\nUm\njust a second then.\nOkay. So part of what we are doing in\nterms of scaling the ability to uh\nhandle complexity is putting in the\narchitecture things that uh try to do\nsmaller thing in a much more isolated\nmanner. So that's uh uh you know\nwhenever you have a lot of con uh too\nbig of a context window or too large of\na history of messages you might run into\nsome problems. Other mechanisms that we\nhave here is a hierarchical short\nlisting of uh tools. So we have\nmechanisms that kind of uh uh try to ask\nyou know what tools are available um and\nand match them to the context of the\ntask at that point of time only like the\nname and the description of the of the\ntool uh you know kind of is brought back\ninto a second reasoner model that now\nwill decide to select one or more of\nthese to accomplish the task for which\nyou will get now more information from\nthe uh index about the full schema the\ninputs and the outputs of of specific\ntools. Um we are also working in areas\nuh that will make coua remember from\npast experience. This is not yet in the\nrepo but uh actually coming up really\nreally soon so that coua can improve\nover time and when it remember uh like\nthe relationship between uh tasks and\neventually the selected tools and um uh\nthe success of the trajectory it will\nderive some uh we call them sometimes\ntips sometimes guidelines uh from that\nso that the second time you can actually\njump and and skip a lot of the reasoning\na lot of the uh searching and just get\nto executing the um the tools that were\ncreated. This actually has also the side\nbenefit of consistency because now you\nrun tools and less of LLM. And this is\nsomething that our customers really\nappreciate because uh you know the\ncreativity of an LLM to generate every\ntime something even if it's the\n[clears throat] the right data but the\nstyling of it is different you know it\nloses in terms of uh the trustworthiness\nby users. So to be consistent, we are\nalso adding uh mechanisms for that.\nYeah. No, that makes a lot of sense. I\nthink you make two really good points.\nLike one, like tool names and tool\ndescriptions are like brittle. Like it's\nnot just just feeding those into agent\ncontext is like sometimes not enough,\nespecially if you have like thousands of\ntools, right? So I think that's really\ncool that using sort like a different\nmodel, a different kind of maybe sub\nagent for selecting tools. And then um\nin terms of like yeah like learning from\nexperience of the agent that's also\nreally cool um like it's kind of like um\nyeah adding like specific instances or\nkind of like adding to the like the\nprompt or the instructions of the agent\nas it experiences things. So that's also\nreally interesting. Um\num I guess yeah so we don't have a lot\nof questions today. Um but um yeah I\nthink that'll I think that'll do it for\nus. If anyone has any questions, um, put\nthem in the chat now. Um, that'd be\ngreat. Um, maybe we'll give a couple\nto reach out to us and to go to a repo\nand try it out and, uh, you know, star\nus. That would also help us. Uh, if you\nlike the repo and um, yeah, we are we\nare here not far away.\nCool. Okay. All right. That'll do it.\nThank you. Thank you so much, guys. And\nthank you everyone for attending as\nwell.\nIt's been a pleasure. Thank you.\nBye-bye.",
  "transcript_chars": 19095,
  "ingested_at": "2026-05-12T00:39:37.966859+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 181,
    "like_count": null,
    "channel_id": "UCrVHzD-psX5IMCGoEWHXmGw",
    "categories": [
      "Entertainment"
    ],
    "tags": []
  }
}