{
  "video_id": "D_FCYsshMI4",
  "channel_slug": "openai",
  "channel_handle": "openai",
  "title": "Computer use in Codex",
  "duration_seconds": 686.0,
  "url": "https://www.youtube.com/watch?v=D_FCYsshMI4",
  "upload_date": "",
  "transcript": "And we'll roll in both cameras.\nGreat. Thank you.\nYeah.\nHi everyone, Roma here. Codex has\nquickly evolved from a coding agent into\na real teammate. But not just a coding\nteammate anymore. You can literally use\nCodex for any tasks and computer use is\na big part of that shift. It takes Codex\nbeyond your tools and files and into the\nreal work you do with your local apps.\nToday I'm joined by Ari who has spent a\nlot of time thinking about this problem.\nSo Arie, why computer use? Tell me more\nabout how this works.\nYeah, I'm so excited about computer use.\nCodeex already had the ability to do so\nmany things on your computer because it\ncould run commands. It could write code.\nSo it can solve all kinds of problems\nfor you. What's new is that there's all\nthis software on your computer that is a\ngraphical user interface. It's sort of\nsomething that as a human you use by\nlooking at it, by moving your mouse, by\nclicking, by typing. And now Codeex can\ndo all of that for you also. So it can\nuse literally any application on your\ncomputer which is so powerful. It's\nreally exciting to get to sort of see\nthis come together and and make\nsomething that people can use for so\nmany different things.\nOne of the things that I that I found\nvery delightful was the on boarding. So\nfor people watching this who want to get\nstarted with codeex and want to try\nthese like amazing features, the very\nfirst onboarding screen is very easy,\nright? Do you want to show it to us?\nYeah, I'd love to. Yeah. Let's say this\nis the first time I'm using computer\nuse. It's going to ask for my permission\nfirst, right?\nAnd so when it does that, I'll get this\nwindow that says enable codeex computer\nuse. And when I press allow,\nit animates the panel straight into the\nsettings window, which just helps helps\nyou know where to look and and what\nyou're supposed to do next.\nIt tells you how to drag it\nand it tells you how to drag the list.\nAnd then you have to authorize because\nyou're making changes to your system\nsettings. And now, um, I was able to set\nup the whole thing in two drags. And\nthen now you can see it's going and\nclicking and and doing the task for me\nand it's now done.\nAmazing.\nYeah.\nCool. So let's see computers in action.\nUm do you have one task on top of your\nhead that you want to show us?\nYeah, absolutely. So one thing that I do\nevery so often is I need to test\nsoftware in older Mac operating systems.\nAnd so for that I use virtual machines.\nAnd I have an app I love called UTM. But\nit's a pain to create a virtual machine.\nI have to click through a bunch of\nthings. I have to run the Mac OS setup\nassistant. And so, um,\nsounds like a perfect use case.\nPerfect use case. So, now I can save a\nwhole bunch of time by having the agent\ndo it for me. So, I'm going to go into\ncodeex and I'm going to say make a new\nMac VM in UTM. And so, when I type at,\nit shows me a list of the apps I have on\nmy computer and I can run the query and\nthen it'll actually um start using the\nthe app I selected. So, um, in this\ncase, it's going to spin up UTM. And\nwhat we can see here is once it starts\nthe app and once it starts using the\napp, you'll see the cursor fly in.\nThat's awesome.\nIt's so cool. What's cool about it is\nthat it's different from my cursor. So\nCodeex can click around without\ninterrupting what I'm doing on my\ncomputer.\nSo you can keep on using your computer\nwhile Codeex is working in the\nbackground.\nYeah, that's exactly right. You know, a\nlot of computer use implementations, in\nfact, every computer use implementation\nI've ever seen, takes over your entire\ncomputer. So, you can't use your\ncomputer while the agent is using your\napps.\nAnd now it's already done. It sounds\nlike it's downloading Mac OS.\nIt's downloading Mac OS. So, um you\nknow, once Mac OS finishes downloading,\nit can also complete the next step,\nwhich is actually um setting up Mac OS\nfor me.\nUm which so much time.\nShould we like uh try to do another uh\nof these computer use like tasks in the\nbackground? Can you do multiple?\nAbsolutely. So,\nI want to focus play some good music for\nuh for work for me in in Spotify. So,\nnow the agent's going to start using\nSpotify. Um, but what's super powerful\nabout this is it can actually do things\num across multiple applications. It can\ndo multiple curses in multiple apps at\nthe same time. So, I'm going to say\nadd a reminder in the reminders app to\nuh tonight to look through my tax\ndocuments.\nThe music's coming.\nMusic's going in. Spotify\nstarting to add some reminders for me.\nUm, so now all of a sudden my Mac is\nthis multitasking environment where I\ncan do uh so many things at once and\nhave agents do all the things that I\ndon't want to be spending my time on.\nYeah.\nOkay. That's that's so cool. Now you\nhave like three apps in the background\nthat Codeex has been driving and\neverything you've done with the cursor\nhas been so delightful too. Like do you\nwant to tell us more about this? Yeah,\nwe wanted to make something that felt\nfun to use, that felt natural. And so\nthe motion of the cursor is something\nthat is important when you're watching\nit use your apps. You sort of want to\nunderstand what it's doing. And so we\nput some effort into finding these uh\ncurves of motion that feel natural and\nfeel kind of whimsical where the arrow\nturns in the direction of motion so it\nlooks like it's swimming across your\nscreen. It's makes it fun to use.\nYeah, it's really delightful also to\nlike have a better sense and\nunderstanding of what the agent is\nactually doing with every one of these\napps.\nYeah, I really love it. Um, one thing\nthat I want to touch on is you can use\ncomputer use with a faster model like\nSpark, right? Tell me more about how you\nguys start about multimodal and\naccessibility combined.\nYeah, we've put together some really\nexciting things uh with the way that it\nworks with the model and it's just only\nthe beginning of the kind of work that\nwe're able to do. Historically, computer\nuse has been only something that works\nwith screenshots. It takes advantage of\nthe power of multimodal models. So the\nmodel can see the interface and and\nclick and type by coordinate. Um, which\nis is great, but it turns out there's\nall this hidden information that is\npossible to extract about the interface\nof an application through the\naccessibility framework. And so we have\nspent a lot of time figuring out how to\nmake use of this in a way that enhances\nthe model's abilities. We pull a bunch\nof information that is textual\ndescribing the interface and the model\ncan use that to see things even that are\nscrolled off screen. it can understand\nmore deeply the role of each element\nthat's on screen. And so this just makes\nthe model super accurate at performing\ntasks. And then the other benefit of it\nwhich you were alluding to is that\nbecause it doesn't require images\nnecessarily, we can use non- multimodal\nmodels like codec spark which are super\nfast. And so all of a sudden you have\nthis experience where computer use can\nwhen you use one of those models use use\nsoftware even faster than you can.\nThat's amazing. Do you want to try do\nyou want to try one of these tasks for\ninstance? were like we would switch the\nmodel to Spark.\nYeah, absolutely.\nRelied remain in messages to try\ncomputer use for debugging apps. And so\nwhat we'll see is before computer use\nwas, you know, pretty performant, but\nnow with this with the Spark model, it's\nlike super human. It uses the software\nliterally faster than than than a human\nwould. We see it here like open the text\nuh type the message now to me\nand in a second it sent\nthat's pretty incredible.\nPretty sick. So it just did this in the\nbackground. I was able to do other\nthings on my computer at the same time\nand it's super\nand we got it.\nI have it.\nVery nice.\nAsking me to try computer use for\ndebugging apps.\nSick.\nIncredible. You brought so much from\nyour knowledge of Sky into the Codex\napp. Now this is incredible. U working\nwith the research team now at OpenAI.\nWhere do you see the the future uh of\ncomputer use?\nYeah, earlier products like operator and\nchatbt agent for those products we used\nto train dedicated models for computer\nuse. And since then the research team\nhas done this amazing work to actually\nbring those capabilities into the main\nGPT models. And so now we're actually\nbuilding this on codeex on the same\nmodels that are available through the\nAPI and everyone can build these amazing\ncomputer use capabilities. So that's\nbeen um super super nice and also great\nstreamlining for our workflow\ninternally. I think that, you know, it's\namazing how fast we've been able to get\nthis to work, um, you know, with the\nmainline models and with Spark. Um, but\nI think we are going to want to get to a\nplace where computer use is superhuman.\nYou know, I think that we can get to a\nplace where computer use can operate a\ncomputer two, five, 10 times as fast as\na person. And I think that's where it's\ngoing to become indispensable. You know,\nyou're going to want to use it um for\nfor so many computing tasks, for really\neverything you do in your life. and it's\ngoing to save you so much time and let\nyou focus on the things that are\nimportant. Um, and so I'm I'm really\nexcited about what the road map looks\nlike there.\nOne thing I wanted to touch on maybe\nthat people might be curious about is\nlike the safety approach to all of this,\nyou know, like you have these amazing\ncapabilities for Codeex to now kind of\ndrive some apps on your Mac. How are you\nguys all thinking about like safety?\nYeah, it's such a good question. And I\nfeel like the this type of technology\nhas the potential to be kind of scary\nbecause it's actually taking over, you\nknow, the actions that you would do on\nyour computer and it has access to so\nmuch stuff. So we feel like it's so\nimportant that people feel really\ncomfortable using this technology and so\nwe've been spend a lot of time thinking\nabout how to do that. One of the things\nthat we've done here is we've made\ncomputer use such that it can only\naccess applications that you allow.\nEvery time Codeex goes to use an app for\nthe first time, it asks for your\npermission, right? And when you say yes,\nCodeex can see and type into that app,\nbut it can't see or interact with any\nother app on your computer. So, if you\nhave some stuff that's, you know, maybe\na little bit sensitive um in in one of\nyour applications, you can feel very\nconfident knowing that Codeex can access\nyour developer applications and your\nproductivity applications without\naccessing, you know, anything that's\nthat's more sensitive. And so, that just\nbuilds a lot of trust, I think, for the\nuser.\nAbsolutely. Yeah, that's pretty amazing\ncuz it's not like streaming your entire\ndesktop or accessing all of your files,\nanything like that. It's very much like\ncase by case, app by app. As you're\ntrying to be productive, you're giving\nCodeex the permission to do so. I mean,\nobviously this is a simple task, but now\nthat we've seen the power of kind of\ncomputer use, I'm curious like what have\nyou used uh comput?\nUm what were kind of your magical\nmoments that you've experienced with it?\nYeah, I have all these spreadsheets that\nI use for like financial tracking and\nnow I actually ask Codex to update them\nfor me and I don't have to do it myself\nanymore. Super super powerful.\nThat's incredible. I mean, it's hard to\neven imagine these days starting a task\nwithout Codex.\nYeah, that's so true. Like nowadays when\nI want to start something new, whether\nit's programming or or even something\nelse on my computer, I I feel like I\nwant to turn to Codeex first because it\nsaves me so much time. and we had the\nfile system, we had the plugins to\naccess all of these services online. It\nfeels like the missing piece was\ncomputer use to be able to access the\nlocal apps.\nI definitely think so. Especially for\nme, I use a really wide variety of\napplications. I use a lot of web\napplications. I use a lot of Apple\nnative apps. I actually track my\nspreadsheets in the numbers app. And so\nnow this just brings all of that online,\nall of it into a place where Codex can\nuh can access it end to end.\nPretty incredible. Thanks, Harry.\nComputer news is one of those\ncapabilities that are hard to fully\nappreciate until you try it. All of the\nsudden, your computer works in a whole\nnew way. And it's not just codecs moving\naround your computer. It's Codex\nactually doing real work for you in the\nbackground without breaking your flow.\nSo try it on your hardest task. Maybe\nthe one that has you bouncing around\nfive apps and eats multiple hours of\nyour day. We genuinely can't wait to see\nwhat you think.\nComputer use is available for a Mac\ntoday and we cannot wait to bring it to\nWindows users very soon. Thank you so\nmuch Harry. See you next time.",
  "transcript_chars": 12726,
  "ingested_at": "2026-05-15T10:38:09.044225+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 108183,
    "like_count": 4283,
    "channel_id": "UCXZCJLdBC09xxGZ6gcdrc6A",
    "categories": [
      "Science & Technology"
    ],
    "tags": []
  }
}