{
  "video_id": "XTQTJoSfeMg",
  "channel_slug": "techwithtim",
  "channel_handle": "Tech With Tim",
  "title": "AI Web Scraping Is Insanely Good | Browserbase Full Tutorial",
  "duration_seconds": 1256,
  "url": "https://www.youtube.com/watch?v=XTQTJoSfeMg",
  "upload_date": "20260515",
  "transcript": "If you've ever tried to connect an AI agent\nto some kind of browser or to automate the web,\nthen you've probably dealt with a lot of frustration\nand a lot of problems.\nNow, in this video,\nI'm going to show you the fix to that\nby showing you a platform called Browserbase,\nwhich is free to use.\nNow this allows you to connect your\nAI agent to a remote browser session\nthat can automatically search the web, extract\ninformation, press button, sign in to websites\npretty much anything that you need\nin a remote fashion,\nwhich means if you're running it from open claw\nor a Hermes agent, or a Claude code session, or\nan environment where you don't have your own browser\nor you want to scale up and use, you know,\ntens of thousands of browsers at the same time,\nyou can do it with this platform.\nIt's going to be very valuable to you.\nAnd even if you're someone who does web scraping,\nyou're going to see a much better approach here\nthan manually writing a ton of playwright\nor selenium code, or trying to fix\nall of the CSS selectors on the page.\nSo if you're interested in web automation\nat all, stick around,\nbecause this is definitely something you need to know\nabout.\nSo effectively, what I'm going to do\nhere is just give you a full tutorial\non how to use the browser based platform.\nNow I've partnered with them on this video,\nbut like I mentioned, it is free to use.\nThey have a generous free tier\nand then of course if you use it a lot\nyou can upgrade and buy additional credits.\nNow, browser based effectively\nallows you to control remote browser sessions\nor even your local session as well.\nIf you wanted to, using a significantly better\nkind of API\nor SDK or whatever you want to call it,\nthan something like Playwright or Selenium.\nIf you're not familiar with those, those are more\nconventional kind of browser automation frameworks\nthat allow you to like select a bot in or select\nsome text or search for like a CSS selector.\nBut it doesn't use any type of.\nI just like manual browser automation.\nBrowser based has all of that functionality,\nplus an AI layer,\nwhich allows you to give natural language\nand then perform\nall of the scraping in a repetitive way\nthat's scalable.\nSo you can search, you can fetch, right?\nThere's a browser you can just give to your\nAI agents, which we're going to look at.\nAnd effectively this allows you\nto reliable web automation because it fixes\na lot of the problems\nwith that traditional scraping approach.\nNow just to explain this a little bit better,\nI want to go through kind of a few quick slides.\nYou understand what it is that we're about to do.\nNow first things first.\nWhen we talk about LLMs or agents, right.\nOpenClaw Agent Hermes agent, your Claude code agent.\nThose can usually connect to your own browser. Right.\nBut that assumes, first of all,\nthat you're running on a machine that has a browser.\nSo like a mac, for example,\nand not maybe a Linux virtual machine or,\nyou know, a Linux\nVM that you're running in the Claude.\nBut even if it has access to a browser, typically\nthese agents are good at searching for information\nkind of finding something from a page,\nbut they can't really reason or act on the page,\nso they can't go and press a button,\nfor example, sign in.\nIt can't spin up,\nyou know, 100 browsers at the same time.\nAnd that's one of the core issues that this fixes.\nNow also, even if you were to try to do that\nusing the AI agent,\ntypically you're going to run into rate\nlimits captures IP bans blocks,\nand there's no default API\nfor interacting with these types of sites.\nSo that's kind of where browser based comes in.\nAnd it's a platform that gives your agents\na real, legitimate browser\nthat looks like a real human\nso that it can go and automate\nfollowing all of those tasks\nthat are running into those issues.\nNow, there's five main things\nthat I'll go over in this video.\nSome will cover more in depth than others.\nFirst, we have functions effectively.\nThis is a way where you can have your code\nand the browser bundled in the Claude\nat the same time for very low latency.\nThe stagehand SDK.\nThis is an SDK that you can use to control a browser,\neither remote or local,\nusing natural language\nand AI agents or LMS mobile gateway.\nThis is effectively one gateway, so you can choose\nany AI feature that you want to control, the browser\nor any LM that you want to control the browser\nwithout needing a ton of different subscriptions.\nAgent identity this is the way you can sign\nin, get pass, block, solve captchas, etc.\nand then the browser API\neffectively a way that you can spin up\nbrowsers remotely\nand use them from different AI systems.\nAgain, this could be normal web automation.\nWe're using it from\nsomething like Claude code or whatever, right?\nNow let's continue here.\nI just want to show you\nan example of what the stagehand SDK looks like,\nwhich we're going to get into in a minute.\nYou can see that in something\nlike playwright or puppeteer,\nyou would have to manually say, okay,\nlike I want to click this button.\nI want to fill in this form, right.\nAnd you have to know exactly what\nform the location of it.\nYou need to know,\nlike the reference to it on the page.\nAnd those things can change, right?\nLike this button tag could be random characters.\nFor example, if it's like a react component.\nOr it could just change later on.\nAnd it's very finicky and fragile.\nWhereas if we look at stage hand here,\nyou can see that we could just say,\nokay, I want to take an action\nof clicking the button, okay.\nI want to take another action\nof filling in this field.\nOkay. I want to extract this from the page.\nSo you just write in natural language\nwhat you want and the model automatically figures out\nhow to do it on the page, which is self-healing,\nand means if something changes on the page later\non, it can still recover from that, rather\nthan if you manually hardcode all of the selectors.\nNow it really has three main things act, extract\nand then observe.\nThere's also an execute one, which\nwe'll look at in a second, but it's a significantly\neasier interface or API to use,\nwhich is why I'm going to show it to you.\nNow the next thing to go over here\nis something called Agent Identity.\nNow, browser based doesn't attempt to evade the bot\ndetection effectively.\nWhat they did is they partnered up with Cloudflare\nas well as a bunch of different Captcha vendors\nso that they can tell these providers, hey, I'm\nan AI agent, but I'm authorized to view this page.\nNo, I'm not doing anything\nagainst the terms of service, for example.\nSo let me through.\nSo with that, when you use their browser sessions,\nyou don't have to deal with all of the blocks\nor IP bands.\nRate limits captions\nbecause it automatically identifies itself\nas an agent and then goes through on the site.\nWhereas if you tried to run this locally\non your own computer and you spin up, you know,\n20 Chrome tabs that are being automated by an AI,\nyou're very quickly going to be banned\nbecause you don't have this agent identity feature.\nOkay. So that's all kind of the key information\nthat you need to know.\nNow I want to show you how we can actually use this.\nNow the first way to use this is directly from\nyour own AI agent or, you know, harness or whatever.\nSo something like open claw, Harmony's\nagent, clawed code, codex doesn't matter.\nSo what we're going to do here\nis use something called an MC server.\nWe're just going to add that to our AI agent.\nIn this case I'm going to add it to Claude code.\nAnd then it's going to have the ability\nto use browser based to interact with the web\nand to perform all of these automations.\nAnd again, to do it at scale\nwith multiple sessions at the same time.\nSo I'm just going to copy this configuration\nright here.\nYou can find it from this page.\nJust scroll down a little bit.\nI'm going to go over to Claude Code.\nI'm just running this in the terminal.\nYou can use the desktop app too.\nAnd I'm just going to paste this and then tell it\nadd this configuration to my MCP servers.\nAnd then what I'm going to do is just paste\nmy API key as well, which I'll show you how to get.\nBecause if you see here\nit says your browser based API key.\nSo in order to get that\nyou just need to make a free account here.\nJust sign up for a new one.\nI'll leave a link in the description.\nYou don't need to pay.\nAnd then once you go to your browser based dashboard,\nyou'll see a little thing in the left sign here\nthat says API key.\nYou just press on it to copy it,\nand then we can go back and we can just paste it.\nAnd if we paste the key and just tell it,\nhere is the API key to use.\nRight. And just paste the key.\nThen we're good to go.\nNow you also could just\nmanually add this to the MCP config.\nIf you're familiar with that.\nBut for most of you it's probably easier\njust to have Clyde do it for you.\nSo I'm going to paste it without leaking it.\nHit enter and then see if it adds the MCP.\nAll right so that finished.\nAnd if you want to test this out\nand make sure it's working just restart Claude code\nand then type slash MCP.\nIf you do that you should see browser based.\nIf we click in\nwe can see all of the tools that are available to us.\nAnd then we can test it out so we can do something\nlike using the browser based MCP server.\nGo to YouTube, search tech with Tim\nand find his most recent video that was posted.\nOkay, something very basic.\nLet's go ahead and hit enter.\nAnd by the way,\nif you're wondering what I'm using to dictate\nhere, it's a really cool tool called Whisper Flow.\nIf I open it up you can see\nI am an absolute power user of this 112,000 words.\nIt is free to use.\nI have a link to it in the description.\nIt's just the best AI voice dictation tool\nthat makes it a lot faster than typing.\nSo anyways, I'm going to go yes,\nI want to start this up.\nWhat this is going to do is now start a remote\nbrowser session and then navigate to YouTube.\nSo if I go here, you'll see that\nit's actually running a session right now.\nAnd we can view in real time\nwhat it's doing in the browser.\nSo we should see that it is doing this. Yeah.\nSo it's actually scrolling down right.\nIt's looking at all of this.\nWe can see the network logs.\nAnd if we go back here I'm just going to go. Yes.\nSo we can start extracting the information\nfrom the page and then hopefully give us the result.\nBut it's pretty cool\nbecause you can actually watch what it's doing.\nYou can take over the browser, right.\nYou can start clicking into stuff\nand you can see all of the information,\nwhich makes it really easy to debug.\nOkay. It's saying it's going to end the session now.\nSo if we go back\nyou can see the session is now ended.\nIt's actually going to give us a full recorded\nreplay of everything that it did,\nas well as all of the pages that it went to.\nSo we can watch it in real time\nand see what happened.\nAnd then you can see most recent video,\nClaude Code Views 4.5\nK posted seven hours ago, and we get the result.\nNow, while this is useful inside of something\nlike Claude code, personally\nI find it much more useful in tools like Open Claw,\nbecause typically you're going to run those on\nsomething like a virtual private server\nwhere you don't actually have your own web browser\nor head full browser on the server.\nAnd in that case,\nwhat you can do is you can just connect it\nto browser based,\nand then it can go and access a browser and use that.\nBecause if you ever tried to use Open Claw\nand you've installed it on a VPN\nand you tried to get some automatic browser\nautomation running, it is a huge pain.\nInstead of doing that,\njust connect it to the MCP server and then you can do\nthe exact same thing you looked at here,\nand you give it those capabilities at scale.\nOkay.\nSo that's pretty cool, right?\nObviously\nit works inside of your own kind of agent harness.\nBut where this becomes more interesting\nis when you actually write the code yourself\nand you start doing automations\ninside of things like Python scripts, right?\nEspecially if you want to do something\na little bit more complex.\nNow, what I'm going to do is just show you\na few demos here of this working and show\nyou kind of how the code operates,\nso you can see how you could write this on your own.\nIf you want to build a web scraping script,\nyou want to have a workflow, you get the idea.\nSo have a few demos and kind of code samples\nthat I'll run you through on my computer here,\nbut I just want to note\nthat this is available for various different SDK.\nSo TypeScript, Python, Java, go, Ruby.\nAnd what I'm going to show you here\nis something called the stagehand SDK.\nNow you can use these browser based browsers\nanywhere that you want.\nYou can connect to it from playwright\nor from selenium.\nBut stagehand is the SDK that we were just using via\nMCP that allows us to have this\nAI driven, repeatable workflow.\nSo you can see that\nyou can choose when you want to write code versus\nwhen you want to use natural language.\nYou can go from AI to repeatable workflows,\nand then you can write it\nonce and run it forever\nwhen you have it inside of this script.\nNow, in order to use this,\nyou just need to install stage.\nAnd in this case I've used UV.\nAnd then you need to provide your browser\nbased API key as well as the model API key.\nIf you don't want to use their model router,\nwhich we'll look at in a second.\nSo let me show you a quick example of how this works.\nSo if we run through this code quickly, you can see\nit's as simple as me just importing stagehand,\nwhich I have installed.\nI have a few variables defined to my browser\nbased API key, the model API key\nfor the lime that I want to use,\nand then the browser based project id.\nNow here I have the ability to run this locally\nor to run it remote.\nSo that's kind of an interesting feature\nof stagehand here is that you can actually run it\nin your own browser, on your own computer,\nwithout needing to pay for anything\nor using their service,\nbecause it's just the SDK that they have provided.\nAnd then you can specify the model\nthat you want to use in the model API key.\nSo if I want to do a remote,\nof course I need to pass my browser base info.\nIf not, then I just say I want to use it locally.\nThen here I just provide the name of the model\nI want to use.\nSo if I was using an OpenAI model,\nI would put in OpenAI API key.\nIf I'm using an anthropic model, I put an anthropic\nAPI key right in my environment variable.\nThen if we scroll down, you can see it's as easy\nas me navigating to a particular page if I want.\nI can get the replay link.\nSo this will allow me to go\nand view the session in real time,\nand then we can just say, hey,\nwe want to extract this information.\nThat's as easy as the API is.\nHere's the ID that we have.\nHere is the instruction okay.\nExtract the number one story from the\nY Combinator front page.\nAnd then this is the schema\nor the format in which I want this data back.\nYou can write this however you want.\nAnd this will\nthen force the model to pipe this into that format.\nSo it's easier for you to parse.\nFor example like result dot data, dot result. Right.\nWe can print that out.\nWe can get all those different properties.\nSo this is kind of a really cool part\nof using stage hand as opposed to having to manually\nwrite it in playwright.\nSo now if I just open this up, let's go.\nYou've run basics\nand let's run it in the Claude for right now.\nYou'll see it saves me a session link.\nSo let's go here and open up the session.\nAnd we can see that it's running.\nIt's opened up Y Combinator.\nAnd then if we give it a second here okay.\nLooks like it finished.\nAnd if we go back we can see the title was a agent.\nCoding is a drop points 179.\nAnd then this is the author\nbecause that is the schema that we provide.\nAnd that's literally as easy as it is to use.\nNow we can also run this locally.\nSo we can say you've run and then local. Same thing.\nWe would just use this configuration here.\nYou're going to get all of this random logging\nbecause that's kind of how it is set up.\nAnd then in a second we should get the top stories\nhere.\nYou can see it's kind of buried in here.\nBut it says top story.\nAnd then we get agenda coding as a trap.\nAnd in this case we didn't run it on browser base.\nWe just ran it on our own computer\nusing the stagehand SDK.\nCool. So that's a basic example.\nNow what I want to do is show you some more advanced\nexamples using some more features from the SDK.\nSo here's another example.\nWe're going to load this up remotely.\nNotice this time\nthat I don't actually provide the model API key.\nAnd the reason for that is I'm using something\nfrom stage hand here called the model gateway.\nNow this allows you just to specify the model\nthat you want to use and have stagehand\nor browser based automatically go\nand then route the request to the appropriate\nalum and just bail you out, whatever the cost is\nthat you would have to pay anyways.\nSo rather than you managing,\nyou know, hundreds of different keys or having a ton\nof different subscriptions, you just literally pick\nthe model you want to use, okay?\nYou want to use OpenAI,\nyou want to use anthropic, whatever.\nIt just does the routing for you\nand you don't need to change anything in the code.\nYou literally just change the model\nthat you want to use.\nNow, alternatively, like we were just doing,\nyou can provide your own API key and do that\nif you want.\nOr you can just use their model routing,\nwhich is now what I'm using in these examples,\nwhich is why I don't have the key.\nSo I say, hey, I want to use sonnet 46, okay,\nI want to use the browser based browser.\nAnd then same thing we can navigate.\nWe're going to go to the Y Combinator page.\nAnd this time I want it to do something\na little bit more complex.\nSo I say extract the top five stories\nfrom the Hacker News front page.\nOkay, we can get all of the stories.\nI then say I want to observe the following.\nSo I want to find the link\nthat opens the comment page for the top story.\nSo what observe will do is it\nwill actually look at this site\nand find anything that you can interact with.\nSo I can find like the comments body and I can find a\nlike body and I can find a search field.\nWhatever you can see here\nidentifies and returns available actions\non the current page that match the given instruction.\nThen we can say, okay, did we find any actions?\nIf we did, we can act on those\nso we can say hey to the action.\nI want to take is I want to use this particular input\nand I want to fill it in, or I want to click it\nor whatever, so we can perform that action\nand then we can extract afterwards.\nSo notice it's just these kind of 3\nor 4 different methods where we are.\nWhat do you call it.\nExtracting right.\nObserving and acting.\nAnd that allows us to use this natural language.\nAnd then of course we can use code as well.\nLike we've kind of done right here.\nBut we have the ability to choose\nwhen we want to do that,\nto get something that's more robust\nbut also a lot easier to write.\nSo if I open this up now,\nI'm going to give you a demo of how this works.\nAnd now you've run demo scraping dot pie.\nOkay.\nLet's go to the replay link\nso we can kind of watch it in real time here.\nAnd let's see what we get okay. So it's on the page.\nWe can see that it's now actually clicked into this\nwhere the comments are right for agent\ndecoding is a trap.\nIf I wanted to,\nI could even tell it to leave a comment.\nOf course we'd have to log in to do that,\nbut if we go here, you can see that\nit found the top five posts, right?\nAgenda coding. Let's by spirit hair,\nall of this stuff.\nIt finds the common thread for story one.\nSo it says act on the link\nto the comment page for the top story.\nSo it's now clicking into it as it just did.\nAnd then it found the top comment.\nInterestingly, I've done whatever blah blah blah.\nAnd if we go back and we watch the recording,\nyou can see that again.\nLet's kind of go here.\nWhere did it click into it?\nFrom here it was able to actually press that,\nand then it was able to scrape the comment,\nof course I can do something much more complex,\nbut I'm just showing you the basic usage of the API\nso that you understand\nhow this actually operates from code.\nAnd again, this is significantly easier than if you\nhad to write this yourself with all of the selectors.\nSo that's some kind of basic scraping task right now\nI want to show you a more advanced workflow,\nbecause sometimes you have pages\nwhere the steps aren't necessarily clear,\nor you want the agent to kind of take on something\nbigger.\nNow, that's what this specific, script does.\nThis actually goes to Google Flights,\nand it punches in a bunch of information,\nwhich for me has always been notoriously difficult\nto automate and then gives me flight results back.\nSo let's quickly run through the code\nand then I'll give you a demo of this working.\nOkay. So same thing. We're connecting to stagehand.\nWe're just picking the model that we want to use,\nwhich in this case is sonnet 46.\nWe said we're going to wait for the capture solve.\nSo it will automatically solve the captures for us\nif it needs to do that\nsame thing, get the session and then view the replay.\nWe're going to navigate to Google Flights.\nWe're just doing some logging.\nThis is what we're searching for San Francisco\nto Dubai on these dates.\nAnd then if we scroll down here notice\nI'm using a new feature from the API called execute.\nNow execute allows us to do multiple complex\nsteps in one flow.\nSo rather than just pressing one button\nor extracting one piece of information,\nwe can do something a lot more complex.\nSo notice that I gave\nthis really detailed instructions on Google Flights.\nSet up the round trip search and click search.\nThis is the type.\nThis is the origin destination, departure, etc.\nafter clicking search,\nwait for the flight results to load and then stop.\nDon't click on any individual flights\nbut I didn't tell it.\nPress this, do this, go here.\nI just gave it a general high level task.\nI said okay, take a maximum 100 steps.\nHere's the model I want to use.\nHere's the timeout which is just a few minutes.\nAnd then it will actually just go and perform\nall of those actions.\nThen after it's done I use extract where I say\nsame thing.\nHere's what I want to extract.\nThe top three flights.\nGive me the information, the,\nyou know, departure date, whatever.\nAnd then it gives me all of the properties here that\nI pipe into the schema, and then I can print it out.\nSo this is 90 lines of code\nto do something that I've done before\nmanually using something like playwright,\nwhich takes a very long time.\nSo let's just run this.\nYou've run and then what is this\ncalled the demo workflow dot Pi.\nAnd let's see it functioning okay.\nSo we can see that it's searching here. Same thing.\nIf we want we can go over to the replay.\nThis one will probably take a little bit\nlonger to run just because it does need to do\nquite a few steps,\nbut we'll be able to watch it here in real time.\nSo I thought it was going wrong here,\nbut it's actually brought itself\nback to the correct page\nand now it's doing the date selection.\nIt's kind of scrolling through the months\nto get into May here.\nAnd okay, there we go.\nIt picks May 8th.\nNow I'm looking for 15th hopefully okay.\nAnd now it's actually fully complete.\nWe have the recording there.\nAnd if we go back here we can see that\nit found the three piece of information.\nSo Air Canada, Emirates and Alaska as we asked.\nAnd it was able to go through\nall of these different steps.\nIt took about four minutes to run.\nBut that makes sense given what's happening here.\nAnd what's actually interesting is now that this will\ncache the result, we can actually set up caching.\nSo if it does that, that means the next time\nwe run this, it will be significantly faster\nbecause it doesn't need to search for all of the\nsame elements because it's already been cached.\nAnd just to give you an idea of what I mean\nin terms of doing this manually,\nthis is an example of a script\nthat uses playwright to do this.\nYou can see the we're already at double\nthe number of lines, but it's not really about that.\nIt's more that we need to actually do things like\na regular expression\ncheck on the price, the duration, the stops.\nI need to get the role of a page.\nI need to find both ends.\nI need to search for various fields,\nand this most likely won't even work\nreliably because things can change on\nthe pages is very dynamic.\nI have to look for Aria labels, right?\nAnd just creating this\nscript is a lot more difficult, if even possible.\nAnd again, many times\nI've tried to automate Google flights.\nIt is very difficult to do\nand this just did it on the first try with me\njust giving some general natural language.\nSo playwright selenium, it's great.\nIt's fine, but only when you really know\nwhat the page is going to look like.\nAnd even so, if the page changes,\nthat's very problematic and is going to be\nvery annoying to deal with.\nSo anyways, guys, that's really\nall I had for you in this video.\nNow there's obviously a lot more stuff\nyou can do here with browser based.\nAnd when you start going up to scale and running\nmultiple sessions, that's when this really does\nmake a lot of sense.\nBut I wanted to give you a quick tutorial\nso you got to understand how to use it,\nwhat the API looks like,\nand to see if it fits your use case.\nIf you guys enjoyed the video\nand you want more content like this, then definitely\nleave a like subscribe\nand I'll see you in the next one.",
  "transcript_chars": 25312,
  "ingested_at": "2026-05-21T19:04:18.957714+00:00",
  "source": "retry-no-transcript",
  "yt_meta": {
    "view_count": 11897,
    "like_count": 383,
    "channel_id": "UC4JX40jDee_tINbkjycV4Sg"
  }
}