{
  "video_id": "_y_ixdk9aRg",
  "title": "Build and Deploy an AI Coding Agent | Cursor Clone with Next.js 16 | Full Course 2026",
  "url": "https://www.youtube.com/watch?v=_y_ixdk9aRg",
  "transcript": "Enter a prompt and watch the AI build. It creates files, modifies components, wires up logic, all in real time. When it's done, hit preview and your app is running right in the browser. This is Polaris, an AI-powered IDE we're building from scratch. In part 1, we set up Authentication, the convex database, and built out the entire editor with syntax highlighting, code folding, minimap, and AI-powered suggestions. Now, in part 2, we're making the AI conversation system actually do things. We're using Ingest Agent Kit to build a tool loop, so the AI can create files, update code, delete folders, and keep iterating until the task is complete. Then there's web containers so you can see your project running live with full terminal output. GitHub integration lets you import existing repositories or push your work to a new one. Clerk handles billing to gate premium features behind a pro plan. And at the end, we're deploying the whole thing to production. And now let's finish this project. Using the link on the screen and coupon code ANTONIO, you can get an extra thousand Firecraw credits for free. We'll be using it to scrape documentation and feed it directly to our AI as contacts throughout this build. If that sounds useful for your project, feel free to grab the deal. In this chapter, we're going to continue working on our conversation system by enhancing it with AI agent and tool execution capabilities. Before we dive into that, we're going to go back and implement what we left over from the previous chapter. We still have to implement message cancellation flow and we have to build the history dialogue. After we do those two, we can go ahead and configure the AI agent with system prompts and create a complete tool execution system. so let's go ahead and make sure our app is running so make sure you have npm run dev make sure you have npx convex dev and make sure you also have npx ingest dash cli dev so you should have three running npm run dev npx convex dev and npx ingest cli at latest dev Great. Now let's go ahead and just test out our app a bit just to see what we can do and what we can't. So if I go ahead and open my app right now, I can definitely send a message, hello world. But one thing I immediately notice is I cannot cancel it, right? So even though I have an indicator to cancel it, there's nothing really happening here. also i don't think there is a possibility to revisit older conversations so i can definitely start a new one new conversation that works too but i cannot revisit the old one we've just tested out and besides that obviously there is no ai processing this is just mock so let's start by giving the ability to cancel a message so this will be very useful to save some tokens right if you want to quickly cancel, you should be able to do that. Same goes if there is a very long running message here and the user in the meantime opens a new conversation and they send a new message, we should also make sure to cancel the previous one because obviously they meant to discard that. So let's go ahead and see how we're going to do that. And I just want to give you one quick tip. If you go inside of your graph here, source control, you probably see that I have these two commits that you never saw me do. I call them override because that's what they are. I just use them to update my readme file. That's it. It's because I finished part one of this tutorial. So I had to update the readme so people can actually see what the project is about. That's it. Your last commit should be this 12 conversation system inside of a merged pull request. So all good. Don't worry about the fact that I have additional commits. It's just a readme change. So the first thing I want to do is I want to implement the route to cancel a message. So I'm going to go inside of source app folder API. And inside of here, we already have messages folder. Perfect. So let's go ahead and simply create a new folder called cancel. And inside a route.ts. now in here let's go ahead and import the following we're going to import z from zod next response from next server and out from clerk next js server let's import ingest client as well as the convex client and then finally let's import api and id type from convex let's quickly define the request schema. So what will this request accept? Very simply, a project ID where the message is currently processing. So we're going to make sure that at any point, only one message can be processing for this project. Now, the reason I'm doing this is simply so in this state of the application, we don't have too many ways to overspend our tokens. Obviously, this can very easily become a limitation if you ever wish to implement something like multiple agents, right? But it's a very easy change. We're not really doing any architectural preventions of multiple running or processing messages. I'm simply implementing this so you don't have accidental running and spending requests. So this will be a post request. So let's go ahead and simply export that and let's extract the user id then we can immediately do an alph check so if the user id is missing let's simply throw a 401 after that we're going to go ahead and extract the body using await request.json and let's immediately parse that using our request schema this will allow us to have a type safe param and also throw an error in case it's invalid. Now let's go ahead and let's get our internal key. So const internal key process.environment and let me go ahead and remind myself of how we actually call this. Do we mention that? Here we do. Here it is. Polaris convex internal key. So I'm just going to use that here again. The exclamation point at the end basically makes sure that this, right now this is string or undefined. If I add an exclamation point, it's just a string. So it's just for type safety. So I have a habit of putting an exclamation point at the end in case you were wondering. But we don't have to do that because we're going to do a check here. If the internal key is missing, we're again going to throw a next response 500 with an error internal key not configured. To give you a quick reminder, we need the internal key because that's the only way we can query a convex client from Next.js API routes. So what we have to do now is we have to find all processing messages in this project. So yes, every time the user hits this cancel endpoint, we're going to make sure that we cancel every single message that's currently processing. So we have at least one button which says, okay, stop all token spend, right? Just stop, abort everything. So we can very easily abort every single running background job, which is very useful. So how do we get processing messages? Well, by awaiting a convex query. The problem is we haven't developed this query. So let's go ahead and do that. So save this file and then we're going to revisit our convex system. .ts. I'm quickly going to check do we have anything called processing messages. We do not. Great. It means we have to develop it. So I'm going to go all the way at the bottom here and I will do export const get processing messages. And this will be a query which will accept arguments of internal key and project ID. And then as usual, we're going to have a handler, which is an asynchronous function, which allows us to access the context and the arguments. First things first, we're going to validate the internal key using a helper we have developed. I think this type error here is just something I have to restart. Yeah, that's it. let me go down to the project to the function I was developing. So validate internal key is just a little helper so we don't have to write this whole thing every single time. It basically checks that the proper and matching internal key was passed. And then from here all we have to do is we have to query the database for messages. So let's go ahead and do that. I'm going to query for messages. And then, because if you take a look inside of our schema.ds in the messages, we have an index called by project status. And so basically, this means not by project status, it means both by a project ID and by messages status. So we can very easily and in an optimized way, using an index, find a message from a specific project which is currently processing, which is exactly what we need right now. So we're going to use the with index helper like this, calling the by project status, which we defined right here. And then we're simply going to query if, let me go ahead and display this like so. if query project ID equals arguments project ID and if status is currently processing and make sure to execute the collect method at the end that's it this will be a system helper which will allow us to at any point in time show us all messages which are currently processing this will almost exactly well should always correlate to us having a running background job So now that we have that function, we can go ahead and go back inside of our new cancel route here, and we can call that specific query. API.system get processing messages. the arguments it accepts is our internal key and project id project id which i'm going to typecast as id projects there we go now that we have the processing messages let's first check if we can easily return already if there are none so if there are no processing messages well let's just break this API method. No need to go any further because we can end here. But if that's not the case, we now have to cancel all the background jobs for each of these messages. How do we cancel a background job? Well, programmatically, we do that the following way. Let me just remind myself a little bit. We have these functions, I think it is inside of features conversations ingest process message here it is instead of our process message ingest function we have developed a cancel on property here and the way we can programmatically cancel a background job is if we send an event message forward slash cancel and if we pass along a message id property And then this syntax right here is going to check if this event's message ID matches process message event data ID. So we have to explicitly know what message ID we want to cancel for. Luckily for us, we have all of that information. So since this can be a lot of processing messages, even though it shouldn't, it should only be one. still I want to develop it in a way that many of them can be cancelled at the same time just in case so you don't have to think of that we're going to do const cancel IDs await promise.all and let's do processing messages.map let's run asynchronous method and very simply let's call ingest.send I mean, let's await ingest.send, message forward slash cancel. And we also have to pass the necessary arguments. So data is going to be message ID, message underscore ID, like that. So now for each of these processing messages, which we have successfully queried, that they have a processing status in the convex database, we are going to attempt to cancel their respective background job because we have to assume that if a message has a processing status, it must have a running background job because the only way a message can stop having a processing status is after process message function actually finishes. Because if you go at the end here, here it is, update assistant message. and this system function calls update message content and in here we set the status to completed. So if there actually is something in the database which is still in processing status, it must mean most likely that there is an equivalent background job running and this way we cancel it. Great. Now that we have that, we should also tell the user that a message has been canceled. So for that, I'm going to call convex.mutation, api.system. And similarly to update message content, we're actually going to implement a new method called update message status. So we are not really interested in anything other than the status. So let's go ahead and quickly implement update message status. I'm going to go back inside of my system.ds. and let me see, can I maybe copy? Yeah, we can copy this entire thing. This will be called update message status. It's going to be a mutation. It will accept an internal key. It will accept a message ID. But for the next argument, it's not going to be content. It's going to be status, which is a union of processing, completed, and canceled. Always double check that that matches your schema. So you should be able to copy it from here and add it here. And it should be exactly the same. I don't think the order matters, but there should be no typos, right? As always, we have to validate the internal key. And after that, we do the patch method, but only on status. and specifically we do it on arguments.status. There we go. That's our update message status. We can now go back inside of the route where we now have the equivalent update message status. We can pass in the internal key message ID to be message underscore ID and the status will be canceled. And finally, return message ID. like that and once we've done all of that let's go ahead and do return next response like so dot json success true canceled true and message ids canceled ids what we return from this api endpoint is completely irrelevant I just want to make it useful. So we successfully did it and we successfully canceled. Do we really need both? I don't know. I think we can make do with just success true. And message IDs is useful just so you can at least see in your network history, all right, how many of these messages were actually canceled by calling this endpoint, right? Great. So that's the backend part finished. What we have to do now is we have to revisit the conversation sidebar. So the conversation sidebar is currently missing a proper handle cancel method. So conversation sidebar located in features, conversations components right here. Let's go ahead and let me just find. Yeah, so nowhere in this code do we have a proper developed handle cancel method. So let's go ahead and do it. const handle cancel is going to be an asynchronous method. So let me fix the typo async. And in here, we're going to open a try and a catch method. In the catch method, we can already throw an error, unable to cancel request. And in the try, we're going to await KY.post API messages cancel. So just make sure you don't misspell this part. because it's not type safe. What I mean by that is that there's nothing stopping you from writing this accidentally. So always double check, you know, is your API actually called that? Messages cancel like this. And in the post, we have to pass in the JSON. The only thing we actually expect is the project ID, which we can use a shorthand alias like this. Perfect. now that we have the handle cancel let's first learn how to call it explicitly and i think the simplest way simplest way is to add it inside of the handle submit and it's this case so if we are currently processing and no new message has been submitted this means that this is just a stop function so now we can actually await handle cancel in here if is processing and if there is no message dot text. So let's see that in action. In fact, I want to make sure that I have my ingest running somewhere. Here it is. All right. And I have this. And I also just for fun, I'm going to make sure to increase the sleep time of my function so we can actually see this effect. So go inside of features, conversations, ingest, process message, and change this from 5 seconds to 50 seconds. Simply so we can actually see the cancel effect happening. So now, I'm going to go ahead and do testing cancellation. And I'm going to send it. And we should have a new background job running here. And this should wait for 50 seconds. but if I click this hopefully and looks like successfully we have canceled it here it is you can see it says canceled and I think there is somehow a way to observe observe this cancel event because technically we did just send a whole new event so it should be documented somewhere. I'm just not sure where, right? But what we just did is we used Convex database to find, let me actually open Convex, perhaps that's going to help us too. So if you go inside of your project here in the Convex dashboard, inside of data, specifically inside of messages here, you should find status column here. And you can see that my last one has canceled status. So just to make things easier, I'm going to delete all of my messages. And I'm going to also delete all of the conversations. Then I'm just going to go ahead and create a new brand new project here. So it's easier for me to understand what's going on. and I'm going to send again testing cancellation. So right now what's happening here is that we have a new message. So go inside of your messages table, and you can see that right now there is a response from the assistant, which is currently in its processing status So this is now spending tokens Well not right now but it will in the future This means AI is thinking of something So if user changes their mind and if they hit this what we do is we query all of those messages which are currently processing and we fire a cancel event on their ingest background job. And then we also change their status to canceled, right? so one thing I wish we could do as well inside of the ingest perhaps we can somehow I wish I could change the actual cancel status in here in the cancel on maybe there kind of is a way to do that I'm not sure I have to check documentation but right now what we're doing is we are doing it directly in the API route of the cancel right here so we simply fetch all the processing messages from the database which basically means all the assistant messages that are currently processing we early return if there are none but if there are we send ingest message forward slash cancel because that's exactly what we defined here and we send message id so what's important is that you didn't misspell message id anywhere in here right and also when you invoke process message. Let me see. Process message. Where do I define the event? Here it is, message forward slash sent. So when you invoke this in source app API messages, it's also important that you didn't misspell the variable here. All of those variables are important for this clause right here, which will allow us to successfully cancel a message. Perfect. So So explicitly canceling a message seems to work just fine. The problem is, user has no idea what just happened. Well, they technically do, right? But it would be nice if there was an explicit way to show to the user, hey, this message has no content, and that's because it was canceled. So for that, we should go ahead and find, I think it's also inside of the conversation sidebar. let me find it right here where we iterate over our conversation messages we should check if a message has been cancelled right now instead of the message content here we have if message status is processing show thinking otherwise show the content so we should do kind of a double otherwise. I'm sure there was better ways to say that, but yeah, I think you know what I mean. And in here, let's just go ahead and check. Otherwise, if message.status is equal to canceled, then go ahead and render a span request canceled. and let's simply give it a class name text muted foreground and italic and then in here let's go ahead and make sure we return a message response I hope this is proper syntax, it is, there we go you can see it says request cancel so if I go ahead and send another message here you can see that right now it's thinking and when I hit cancel it will change to request cancelled and allow us to send a new one so we don't allow the user to spam and have a bunch of background jobs running great so now that we have that finished there is still another way a user can have both so if i do this for example if i go uh in this convo i am processing so i will send this message this is now processing right you can see that a message is processing a background job is running and if i attempt to send anything else here i'm going to cancel it but if i open a new one and do hello now we don't actually cancel anything we just have two processing messages and two running background jobs chances are user probably doesn't care about what is in the other one maybe in the future your users will and you will change this behavior but i feel like most of the time this is just user deciding to start a new conversation so it would be a good idea that every single time we send a new message, we cancel all the currently running requests for this project. So let's go ahead and do that. We're going to revisit our source app API messages route.ts. And basically what we have to do is we have this to do called invoke ingest the process the message, but I think we are already doing that, so I think we can remove this to-do. We're just missing some data, which we're going to have to update. But this is the to-do I care about. Check for processing messages. So inside of this async post function here, let's go ahead and find the place where we define the project ID before we call convex mutation create message. Before we do any of that, we're going to go ahead and check for processing messages. Lucky for us, we can now do this quite easily. So in fact, you can open your cancel right here. And you can go ahead and do this, right? So instead of this to do, copy this, paste it, find all processing messages in this project. So we are calling the same function, get processing messages. And I don't think we have to cast it this time. Yes, we don't have to. We can just use this. What's important is that you pass the internal key along and the project ID. And now we're going to check. If processing messages.length is larger than zero, meaning we have some processing messages in this project while the user is trying to send a new one, let's cancel all of those processing messages. and we can actually copy this entire thing. So this await promise all can be copied entirely. Like this and just paste it here. I'm just going to fix the indentation. There we go. So await promise all processing messages dot map. We get the individual message. We first triggered the cancel event from ingest. So we shut down the background job using the equivalent message ID. And after that, we update the message status to cancel. Why? Well, because remember, we can't do that inside of the cancel event. So we have to do it here. And we actually don't have to return the message ID. Previously, we do that because we want to return the network request. But in here, we don't have to do anything like that. And I think that's all we really need to do. everything else I think works just fine. So here we create the user message. In here we create the assistant message which is set to processing. Let me just go ahead and add a comment so we understand what we're doing here. In here we add a trigger to process the message and then we just return like hey we're doing something. If you want to you can also add a little boolean like did cancel any messages and then you can return did cancel any messages true simply saying your network history you are aware that what happened here was some messages were canceled before a new one was created so let's take a look at our situation now i'm going to do the same thing i'm going to start a completely new conversation and i will say in this conversation i am running right so now i want to take a look at here one processing message one running background job and now i'm going to open a new conversation previously the problem was if i sent something i would have two of them running this should cancel the old one so right now nothing should be different here but you can see this one was automatically canceled only the newest one is processing. And this background job was canceled. Only the newest one is processing. And now I can explicitly cancel this one too. And there we go. We now have a very safe way of sending messages without polluting our background jobs, without spending too many tokens. Because remember, all of these are going to be very complex AI requests. You can't really afford to have a leak in your processing somewhere. That's why we are focusing on this so much. Excellent. So with just a few system functions and just a few routes, we created a very thorough system for keeping track that only one message for a project is currently processing. What we ought to do next is implement the history dialogue because at the moment there is no way to revisit previous conversations. Let's get started by creating the past conversations dialogue component. I'm going to go inside of source, features, conversations, components. And in here, I'm going to create a new file, past-conversations-dialogue.tsx. I'm going to go ahead and import, start with a directive, use clients, and then I'm going to import format distance to now. I'm going to add the following elements from the command component dialog empty group input item and list. I'm going to add if we have it let me quickly check use conversations hook from hooks use conversations and I'm going to import id from convex generated data model. Let's go ahead and define the interface past conversations dialog props, which will accept the project ID for which we are going to load the conversations, open, on open change, and on select, which will allow us to select a conversation. Now that we have those props we can define our component like so Make sure to extract project ID open on open change and on select Now let load all conversations for a project ID that the user has passed Let implement a very simple handle select method which will accept a conversation ID called the on select which we can we might as well make on select required and then we don't have to do this weird optional chain. And make sure to close this dialog after the user selects an older conversation. Now let's go ahead and return. What we're going to return is the command dialog component. The command dialog component will have the following props. Open and on open change. As well as a title and description. The title will say past conversations. and the description will say search and select a past conversation. Now we have to define the command input with a placeholder, which will be search for conversations. Then we have to add a command list and we have to add a command empty. This will serve as a placeholder if something the user wrote does not exist in our database, so no conversations found. Finally, let's go ahead and add the command group component with a heading conversations. Now we can go ahead and iterate over our conversations using conversations.map. Make sure to add a question mark since conversations can be undefined. Now in here, we're going to return a command item. the command item will have a key of conversation underscore id it will have a value which will combine the conversations title with the id this is to prevent multiple conversations from being selected at the same time in case they have the same title and let's pass in our handle select right here. Now in here I see I have an error so let me quickly see what that's about. I think I'm missing a parenthesis. There we go. So I was missing one parenthesis here. Instead of command item I'm going to open a div. I'm going to render span inside with a conversation title and I'm going to show the created at timestamp with another span. And instead of using created at, we will simply use underscore creation time because that's built in with convex. So the div has flex, flex column, and the gap 0.5. The span for the title has no styles, whereas the span for the timestamp has an extra small text and kind of a muted color for the text. Perfect. So that is our command dialog finished. And now we have to find a way to render it. So we're going to go ahead back inside of the conversations sidebar, which I believe is, yeah, it's in the exact same folder. I didn't even have to close it. And the first thing we're going to do is we're going to define the state which is going to control whether this is open or not. So let's add it right here underneath the selected conversation ID. So we're going to call this state past conversations open and set past conversations open. And now let's find a button which is going to toggle that. In our case, that's going to be this, the history icon button. So simply give it an on click, set past conversations open to true. Now, after this button right here, actually, well, it doesn't really matter where we do this, But what I like to do with dialogues is I like to wrap my entire component inside of a fragment. And then I like to render it outside. So past conversations and dialogue like this. You don't have to do it this way. I believe all of these dialogues from Shatzian use React Portal, which moves them to the outer DOM. but I still like to semantically render my components how they are going to appear in the DOM. And this will be kind of above all this content because when I see something rendered inside, I kind of expect it to be shown inside but that's not the scenario for the dialogue. But just to be clear, you can render this wherever you want. It doesn't matter. So what do we need to pass here? We need to pass the project ID so we know exactly what conversations to load. We need to pass the open status. We need to pass on open change. And we also need to pass on select. And the on select will simply call our set selected conversation ID, which we have already utilized in this project. So let's go ahead and try. If I go ahead and click on this button, you can now see that I have a bunch of these previous conversations. and you can see that we can see exactly this test cancellations which we were trying just a moment ago so we have successfully implemented that too at the moment we can't really do any useful search here uh you might see some results like this this is because we are matching the id right because we combine the title and the id of the conversation and this is how it looks like if there is no results found. This isn't too useful right now because they are all named exactly the same. And that brings us to, well, our next step, which is basically implementing the agent. I want to start by revisiting our ingest process message function, as well as our API endpoint to create a message. So instead of API messages, route.ts. Right now, when we trigger this message.send event, we only pass in the message ID because that's the only thing we need. But now let's extend it by passing the conversation ID, project ID, and the actual message. Now we have to strictly type those inside of the actual feature. So features, conversations, ingest, process, message. The message event only accepts this too. So now let's enhance that. Besides the message ID, we will also accept conversation ID. We're also going to accept project ID and finally the actual message. Now that we have a proper message event from here, we will be able to extract everything else we need. But let's go ahead and stop here. And let's install a package which we are going to use to, well, set up the ingest agent kit. So I'm going to go ahead and just do npm install at ingest forward slash agent kit. Like so. I didn't shut down my app. I still have all three running. There is no need to shut down your app at the moment. And once this function has installed, I'm going to go ahead and show you exactly inside of package.json which version I'm using. So for me, it is 0.13.2. And it's also a good idea to use the link on the screen to visit ingest agent kit. in here you can find the actual documentation and you can find the exact quick start that we just did we already have ingest installed and now we also added ingest agent kit and you can see that starting with agent kit 9.0 ingest is a required peer dependency you must install both packages together to ensure proper runtime this is actually a good tip perhaps we should run the whole thing again and if we're going to do that we should shut down the ingest tab so yes i'm going to run this again simply because it's a peer dependency i'm not sure if it's any different if you install them in the same turn let me see is my package json any different this is my ingest agent kit version and this is my ingest version so yeah i mean i would suggest just running the install at the same time like this. After that, go ahead and run npx ingest cli at latest dev again. I don't think there should be any issues. If there are, it's probably version related. You didn't write anything incorrectly. All right, so we have that ready. And now before we explore ingest agent kit any further, let's go ahead inside of source let's go inside of features conversations and in here in the ingest folder I'm going to create a file constants.ts in here we're going to define our system prompts obviously you can change and tune your system prompts to whatever you prefer but these are the ones I'm going to use. You can find this using my source code, which you can find a link for on the screen. It's free. And you can also find it in the assets file, which I usually show you. So go ahead and add the coding agent system prompt. If you're using the source code, simply navigate to the exact file I am in right now. And let's go ahead and also add one more, which is going to be the title generator system prompt. Again, using the link on the screen, you can find these in my source code or you can find them in the public assets repository. Now, in order to prepare for tool calling, we're going to have to add a bunch of system queries and mutations in here in the convex system file. So let's go ahead and have that ready. So later we can just simply query them. We're going to start with a very simple get recent messages. We're going to use this for conversation context.\nPerhaps you can add little comments here. Used for agent conversation context. This can help you understand why you need these system functions. The arguments it accepts are very familiar. Internal key, conversation ID, and then the limit. The limit is basically to decide how much context that we want to give to the AI. Last five messages, last 10 messages, or 100. it, right? Depending on how AI is advanced, you might be able to pass along all of them. After you validate the internal key, you're going to query the messages using the by conversation index and simply passing along arguments.conversationID. Make sure to order them by ascending and make sure to execute the collect. Perfect. And then let's go ahead and simply limit them like this and then slice them. There we go. So that is the first one we have to do. The second one we have to do is a mutation called update conversation title. This will be used to get rid of the annoying new conversation for every single conversation that we have. Basically using the context of the messages, AI will be able to update the conversation title. The arguments in here are similar. So internal key, conversation ID, and the title. And in the handler here, we're going to first, as always, validate the internal key. And then we're going to simply call patch on the conversation ID with the new title. And we're going to refresh the updated at key simply so we know which one was the latest one we worked on. So you can go ahead and if you want, you know, add a comment what this is used for. So used for agent to update conversation title. Great. Next one we need is get project files. So I'm going to go ahead and start preparing this. GetProjectFiles accepts internal key and project ID. The handler will validate the internal key and it will simply call all files by a project ID. As simple as that. So this will be basically used for list files tools. Used for agent list files tool. We are going to have to create tools ourselves, and we're going to have to define what those tools do. So when the user asks, how many files do I have in this project? The agent will call list files tool. And then we're going to make the agent call this query right here, which will send back to the agent all the files in this project. So that's how that's going to work. That is get project files. Now we need get file by ID. So this right here, which I've just pasted. Get file by ID is a query which accepts internal key and file ID. And very simply just returns the file ID. As simple as that. The next one will be update file. So this one is what it says. It's used to update the file. If we instruct the agent to change something in the file, we need to create a tool which will do that. So let's go ahead and start by defining the handler, which as always is going to validate the internal key. We're first going to check if the file even exists so we can return early if it doesn't. And then finally, we're going to call a patch method like this. So the only thing we're going to patch are the content and the updated at. And finally, let's return arguments.fileID. So let's slow down a bit and simply write some comments. So get file by ID will be used for read files tool. The update file will be used for update file tool. and now let's go ahead and go on to the next one which is create file and you can already guess what tool that's going to be using so again let's create the same mutation create file accepts internal key project id name content and a parent id with an which is optional right so if this is inside of a folder, it's going to have a parent ID. Otherwise, it's in the root of the project. And make sure to do the validate internal key. So the first thing we're going to do is we're going to get, we're going, we have to make sure that the agent doesn't accidentally create the same name, the file in the folder, right? So because of that, we have to get all files, which are currently in this files project, in this files parent, or if there is no parent in the root folder. Because remember, when you create new files, I think we can actually try this out. If I call this hello.jsx, and if I try to do another one, hello.jsx, I get an error. We shouldn't be able to do that. And neither should the agent. So that's why we have to, in this create file method in the system, prevent the agent from doing that. So if we can find an existing file with the same name and file type file, we're going to go ahead and throw an error. Hey, this file already exists. You cannot create this. Otherwise, let's simply insert a new file. So inside of context database, insert files, add project ID, name, content, type of file, parent ID, and updated at. And lastly, return file ID. So you've already guessed it. The create file will be used for create file tool. And now we're going to create a very, very similar one. It's just going to be used for bulk creation and it's going to be very useful in fact. So create files, not create file. This one is create files. So let me define the handler. Let me go ahead and fix the indentation and like this. There we go. So what are the arguments? Internal key, project ID, parent ID, and then an array of files, which is an object. I mean, each of the item in the array is an object which has a name and content inside. So basically, we can give AI 50 files to be created, but it can only be for a specific folder. So that's kind of the limitation. We can already go ahead and add a little comment here. The create files will be used for agents. And let's add bulk create files tool, because that's what it is. It's used to create files in bulk. We also have to be careful here, right? So let's make sure that there are no existing files with the same name. Now, the way we can do that is a little bit complicated. We're going to define an empty array and we're going to give it a very specific type. So the results is by default an empty array. And inside of here, we expect objects with name property, file id property, and an optional error property, because any of these files in the array can be problematic. So let's start by going over them. For file of arguments.file, first thing we're going to do is we're going to check if there is an existing file with the same name and the same type. If there is an existing file, we're simply going to push to the results array, the name of the file, the ID of the existing file, and the error file already exists. And let's continue because we have more files to create. And we can very easily just create a file otherwise. So insert into files, project ID, name, content, type, parent ID, and updated at. And finally, let's do results.push name file.name and file ID and make sure to return results. So that tool is used to create files in bulk. Now let's go ahead and develop the create folder method. So the create folder method is almost identical to the create file one. Let's go ahead and copy this. Let's go to the bottom. And let's change this used for agent create folder tool. So this will be create folder. And it's just not going to have content. So it will have the internal key, project ID, name and parent ID. Then we're going to validate the internal key as always. And we're going to query files. so by project parent all of this is cool but we're just going to change the existing query to check by file type folder and it's going to throw folder already exists the content in here will be empty and the type will be folder like that now if you want to you can create the equivalent create folders, but more often than not, it's not that useful. I've seen agents call create files way more than create folders. If you want to, you can create it. In fact, it will be a nice challenge for you to create a tool that I don't write simply so you learn how to create different tools All right Now we have only two left So next one is rename file which is used to allow the agent to rename a file if the user requests so. Internal key, file ID, and new name as the arguments. Then a check if that file actually exists. And then again, we have to check. We have to check if a file with the new name the agent wants to rename it to exists in the same parent folder. Same with creating a file. So let's get all the siblings using query files by project parent index querying by project ID and by file dot parent ID and collect. And now we can go ahead and check if it exists. And we have to check if the sibling name matches the new name. We have to check if the sibling type matches the file type. And we have to, of course, check that we are not comparing with the file we are intending to rename, right? Because there is always one file with the same name. We should allow the user to rename to the same file. I mean, the agent in this case. so this will take care of both folders and files great if existing ends up being true we have to throw an error so a file or folder named whatever it's named already exists as simple as that and then let's go ahead and patch this arguments file id name arguments name updated at date dot now. And finally, return arguments file ID. To be consistent, I'm going to add a little comment here, simply so I know that all of these ones are used for agent tools. So this one would be for rename file tool. There is only last one we have to do, which is the delete file mutation. So I'm going to go ahead and define it here. Can I quickly copy the comment here and just change this to delete file tool? There we go. It will accept internal key and file ID and after we validate the internal key let's check if the file even exists in the first place and then depending if this is a folder, we have to recursively delete file folder and all of its descendants. So let's go ahead and define a constant delete recursive. It's going to be an asynchronous function which accepts file ID to be a type of arguments.fileID. Then inside, let's first fetch the item. item await context database get file id if we failed to fetch it let's break this function then let's check if it's a folder delete all the children first so if item dot type is equal to a folder first let's query all the children by using by project parent query using the file id that's currently passed here. There we go. Once we have the children, we're going to delete them using the very same method. So for child of children, call itself. It's a recursive method. And this way, if we encounter another folder, it's going to do the same until all files are deleted. So that's why we're developing a recursive method. now we also have to delete the storage file if it exists we currently don't really use this at all but it's important to not forget that we will have a way to store binary files we will see this in action once we implement github imports so if the file was a binary file we should delete its equivalent storage key. Otherwise, it's just going to be taking up space. And then finally, let's delete the file or folder itself. And then we actually have to call that function for the first time. So await delete recursive with arguments file ID that the agent provided. And let's return arguments file ID. This way, we are completely ready to develop our execution tools. we don't have to worry about revisiting this file again for agent purposes. At least, I'm pretty sure this is all that we need. Most of these are pretty similar. You can always visit the source code if you're unsure or if you think you've made a mistake or if something doesn't work. Great. Now that we have this developed, let's actually go ahead and develop the agent. So the work we have developed now is inside of process message. So inside of conversations, ingest process message.ts. So far, what we have in here is a cancel event. We have a proper on failure if something goes wrong, but we don't actually have a real message.send event. You can see we are just pretending to do some AI processing here. So let's go ahead and change that. Let's actually start doing everything that we need. Let's start by destructuring the proper items. so it's no longer going to be just message id it's now conversation id project id message id and the actual message the internal key check can stay the same that's good now we don't have to delete sleep what i would do is wait for database sync and give it maybe one or maximum five seconds and to do i'm going to add check if this is needed. I'm doing this because during my development, I've encountered some kind of out of sync state where an ingest agent can run faster than convex database updates. And that kind of puts it in a weird position. I'm 99% sure this is not needed. But just to stay true to my original source code, I will show you that I had this and we're going to try and remove it later and then we're going to see what happens. All right. So the first thing we have to do is we have to change the conversation's title. So get conversation for a title generation check. And now let's go ahead and do the following. So we're defining a new step. Basically, we're going to get the conversation using a step. So away step dot run, which we're going to call get conversation. and in here we are very simply going to return await convex.query api.system get conversation by id pass in the internal key and the conversation id this way we can use the conversation to see well first of all does it exist if it doesn't let's throw a non-retriable error It's not found. There is no need to retry any steps, right? Now that we have the conversation ID, we can also fetch recent messages for conversation context. So again, another step we are defining here. Get recent messages. And inside of here, we are going to call our API system get recent messages query with the internal key conversation ID, and we're going to limit it to 10 messages. Of course, depending on how good AI models get or more precisely how cheap AI models get, you might increase this. But the more context you give it, the worse results it actually gives you right now. And it's just more expensive altogether. So that's why 10 is kind of a sweet spot right now. Great, so now we have the entire conversation object. We have the 10 most recent messages inside of this conversation for context. and what we can do now is we can build the system prompt so I'm going to add a little comment here we're now going to build system prompt with conversation history we're going to exclude the current processing message though so let's start by defining the system prompt in a changeable let and we're going to give it coding agent system prompt which we can import from constants Remember, we developed this. I told you you can find it in the source code or in the public assets, along with the title generator system prompt. This is also optimized for anthropic models. It should work just fine with Gemini models too, but anthropic models work very well when it comes to XML. I'm not sure what is the structure for other ones, but I think this should work just fine for all generally. Great. So by default, the system prompt is just the default coding agent system prompt. But what we're going to do now is we're going to attempt to inject the context of the recent messages into the prompt. So first, we're going to filter out the current processing message. So no need for that. We are just interested in the past, right? So context messages, recent messages.filter, and we are simply looking at the message, the current message ID, and we remove it from the last 10 recent messages because we're not interested to adding that into the context. That message will be processed either way. And now we have to check, are there any messages at all before this message? Because maybe it's the first one, right? Now, if it is, we have to create that history text using context messages and then we have to map each message inside of the context messages with the filtered out current message. So each message will return a template literal string, message.role to uppercase, which is basically going to look like this. It's going to look like assistant, how can I help you? And then it's going to be user, do this and this. So that's going to be the history, basically. That's why we're doing this. Message.role colon message.content. That's the structure we are developing right now. And let's also join with backward slash backward slash n. This basically means empty space and I think AI likes it this way And what we going to do now is we going to append to the system prompt all of this history text So system prompt plus equals and I would highly recommend just using the link on the screen to copy this part or using, again, you can use the source code by going directly inside of the process message to just copy this. Of course, you can also pause the screen if you want to write it out, or you can use the public assets folder. Great. Now we have the updated system prompt. Now let's see if we should generate a new title or not. So how do we define that? Well, we should have constants.ts in the convex folder. And in here, you should find default conversation title new conversation. So what we're going to do now here is we're very simply going to check if the current conversation.title matches the default conversation title. Now, you can make sure to just import this from convex constants. Is this the most nice way to do this? There are probably better ways, you know, inside of your schema, we could have nicely added something like is title updated something like that but i'm just using a very quick and easy way to do that also i'm not sure how this works right i can import from convex constants so i assume everything is fine i mean convex is just a folder so yes this should be working just fine this isn't the protected folder or anything but in case this ever causes problems you can always try moving constants somewhere outside and then just having duplicate default conversation title one in convex for the convex functions let's say if i go find in folder and if i search for default conversation title it looks like we're only exporting from here so perhaps we don't even need this here Let me go ahead and actually improve this if I can at the moment. So I want to find all the places where I'm using default conversation title. I can see I'm using it in the conversation sidebar and I'm using it in the process message, but I'm not actually using it in convex. So it makes no sense that this constants file is there. Obviously, I planned on using it there. Maybe I will in the future. For now, what I want to do is I want to move it. and I'm going to move it. So I'm going to copy the constants file and I'm going to set up features conversations right here. So we already have constants inside of the ingest folder. I want to keep that there. These are system prompts, right? So no need to pollute that. So make sure that you have new constants.ts inside of your conversations folder. and inside of your convex delete constants.ts now your app will break so you have to revisit conversation sidebar.tsx and let's go ahead and simply change this to a much more there we go this looks better right inside of the conversation sidebar features conversations components conversation sidebar You can now import that from the near constants, right? We've just added here. Nothing else needs changing in the conversation sidebar. The other problematic spot is the process message, which we are developing right now, which we can again simplify. Let me see, maybe another one back or maybe this one. There we go. So we have two constants, one inside of the ingest folder right here, the other one outside of the ingest folder, but still inside of features conversations. So that's a better place to have that in my opinion. Great. So if we should generate a new title, let's go ahead and create a title agent. So if you're wondering about this create agent and how it works. I highly recommend having agent kit documentation open because I mean this is how I learned how to use it. We're now going to import create agent and anthropic from ingest agent kit. You can of course import google if you're using Gemini. We've already went over how you can define different models. It works very similarly to AI SDK. So let's actually do this. So let's import create agent and anthropic from ingest agent kit. I'm going to go ahead at the top. I'm importing create agent and anthropic from ingest agent kit. Now my title agent here will have a name, title generator. it will have a system prompt of title generator system prompt, which I can import from .slash constants alongside my coding agent system prompt. It is this one right here, the short one. Once I've added the title generator system prompt, I have to define the model that is going to execute this agent. So in my case, that's going to be Anthropic. Again, in your case, this can be whatever you want. It can be OpenAI. It can be Grok. Let me see. Okay, it's Gemini. It's not Google. Keep in mind that, I mean, we're not executing any tools yet, but in the past, Gemini had very bad performance with executing tools. So that might be something you should be aware of simply so you understand that you didn't do anything wrong. Sometimes it might just be the model. If possible, in any way, using Anthropic models would be amazing. But I understand if it's not possible, go ahead and try with Gemini either way. So use a cheap model for title generation. That's why I'm choosing Haiku, because it's fast and it's cheap. For Gemini, I'm not sure what's the equivalent, but you can just use the same model. And you have to add a default parameters. Now, you don't have to do this for every model. For Anthropic, you have to. Otherwise, you have an error. If you switch to, I don't know, Gemini, let's see, does it need it? No, not from ingest. So if you try Gemini, you can see there's no error. But if I use Anthropic, it requires default parameters. So be aware of that too. Temperature is going to be zero. maximum tokens is going to be 50. So we don't need too much power here. We're just generating a title. Now let's go ahead and actually run that agent here. So await title agent dot run, pass in the message and pass in the step. And in here we have the output. From that output, we have to find a text message from the assistant. So the text message can be found using output.find and in that message search if the type is text and if the role is assistant. So we successfully found a response from the assistant in a form of text because the assistant can return tools, executions, a bunch of things. That's why we have to check if the type is text. And now let's do if text message dot type actually is text simply because this can yield undefined right so if it is text let's go ahead and extract the new title which we have to update the conversation for so first things first if type of text message dot content is equal to string in that case let's make sure to to text message dot content dot trim. Otherwise, let's do text message dot content dot map. For each content we have, simply return the text part of that content, join all of it, and then trim it. What we're doing here is we're handling various scenarios in ways AI assistants can respond. You can try and be strict with AI assistants to always tell them to basically do this, but you can never rely on them fully. That's why we're doing the trimming and the joining. And then finally, if we manage to obtain title, let's go ahead and do await step.run update title. Let's do update conversation title. It's going to be an asynchronous method. Let's do await convex.mutation api.system update conversation title which we've developed at the start. Pass in the internal key, conversation ID, and the new title. So you remember, we developed this. Update conversation title used for agent to update the conversation title. Accepts internal key, conversation ID, and very simply, the new title. And we are using it right here to let the agent do the update it needs to do. Perfect. So perhaps we could already try this out. I think, let's see. Let's see if we can try this out. I'm going to refresh my Polaris app here. I'm going to start a brand new conversation and I'm going to try and ask it something specific. Set up a React plus Vite project. I'm going to try something like that. And let's see if it will run this or not. Because I'm not sure we... It's not doing anything. And I think it's because... Let's see. Should generate title. I think all of this passes. I just think that the title agent... Hmm. I'm not 100% sure if it's up to us or if it's up to something else what I'm interested in is the actual ingest server here oh it looks like there is error here oh my balance is too low okay so that's the problem I going to go ahead and just update my anthropic balance and just as I updated my balance you can see that the next one succeeded So let go ahead and look at it again in action So create a React plus VIT project Right now it is named new conversation, but after it processes this, it should change this to react VIT project setup guide. There we go. So our first agent actually works and it works quite well and it's quite cheap and fast. And basically this is a super primitive version of what we're going to build next, which is another agent, but it's going to be a coding agent and it's going to be way more advanced in a sense that, well, it will be able to call and execute tools. So let's go ahead and start doing that. Create the coding agent with file tools. We're going to start by defining the coding agent. And let's go ahead and give it the name. I'm going to call this Polaris. I'm going to give it a description of an expert AI coding assistant. The system will be a system prompt. Remember, we have extended the system prompt with the history text, right, with all the previous messages. So now the system prompt is much better. And now we have to define the model. Again, this is your own choice. Again, Anthropic is really, really impressive when it comes to tool execution. So I'm going to go ahead and use Claude Opus, simply Opus, Opus. I'm not sure how you pronounce it, simply because I found it to be the most impressive. It's also very expensive. So, you know be careful you can also do it with haiku but the better the model is the better the results are going to be right and you need to pass default parameters so temperature 0.3 max tokens 16 000 so these are random limits i found work well you can of course tweak them if you know what you're doing. And in here, we're going to add tools. Right now, we have no tools at all. So how about we create our first tool? I'm trying to think of the simplest one of these we can do. Well, let's just start with a read files tool. That's going to be the one we're going to need. So I will simply close everything that's not process message. And inside of the ingest folder, let's go ahead and create a new folder called tools. And inside read files.ts. Now I'm going to import Zod and create tool from ingest agent kit. Again, I would highly recommend that while you're doing this, you're also reading the documentation on tools. So you are not, you know, just blindly following, but understanding where I found this information and how I know how to write it, right? All right. So once you've imported Zod and create tool, let's also import the convex util, API and ID from convex generated folder. Create an interface, read files tool options to be internal key and a string. and let's also define the params schema here. So we're going to accept file IDs, which is going to be an array of IDs, and we're going to throw errors if it's empty or if the array is empty. So now let's go ahead and let's create the read files tool, which accepts the internal key, and it simply uses the interface we defined above. now let's go ahead and immediately return create tool let's give it the name of read files so that's the name of the tool let's give it a description descriptions are actually quite important the same as the name of the tool so read the content of files from the project returns file contents Now let's go ahead and define the parameters. So the parameters are a z.object, file IDs, and then inside we're going to define an array of string with describe array of file IDs to read, like that. Then we have a handler in which we can get the params, and we can get step, give it an alias of tool step for easier understanding. Let's go ahead and do parsed params schema dot safe parse params. This way it will throw an error if the tool is attempting to fire with weird parameters, if they are not what we expect, right? So if not parsed.success, let's go ahead and return error parsed error issues first in the array message. And this will be sent to the agent. So the agent can then retry even if it fails. So it won't make the same mistake twice. let's destructure the file ids from parsed.data so now we have completely valid file ids at this point or at least we should have let's go ahead and open a try and catch block here and I'm going to do return await tool step question mark dot run read files it's going to be an asynchronous method and in here let's go ahead and define the results. Results by default are going to be an empty array but what we expect here is an array with objects. Each object should have an ID, a name and the content inside. So now that we have that defined let's run them through a for loop. For const file ID of file IDs let's go ahead and fetch the file using await convex query API system get file by ID. Pass in the internal key, pass in file ID, file ID as ID files so we don't have any type errors. If we successfully fetch the file and the file has content, let's do results.push. id file id name file.name content file.content. There we go. Now that we have that defined, let's go ahead and check if results.length is equal to zero. Let's return an error. No file is found with provided IDs. use list files to get valid file IDs. And writing this error message, I realized before we can actually test this out, we will have to write one more function. Sorry about that. But yes, I completely forgot that the agent right now doesn't really know where to pass the file IDs from, right? How is it supposed to know what file IDs to read, right? So this function will actually be used once the agent is instructed to read a specific file ID. So still, we're going to need this function either way. We're just not going to be able to test it out right away. So make sure to return stringified results. And in the catch method here, grab the error and return error reading files. error instance of error display the error message otherwise unknown error like so let me go ahead and check what the okay I should not do something here I should end that here there we go alright so that's our first function implemented now that we have read files tool let's go back instead of process message here and let's go ahead and pass create read files tool you can import it from tool read files and in here simply pass the internal key now technically we could have just as easily use the internal key right here, right? We could have defined const internal key process.environment, blah, blah, blah. But if we pass it as a prop here, it's kind of already validated at this point, you know, simply because we have used it in all of these previous steps and we've checked if it's valid here, right? So I kind of feel like we can just pass it. I don't know. If you feel like this is not a good practice, you can, of course, define it here directly. All right, so that's the create read files tool. Now let's go ahead and implement create list files tool. I'm going to copy read files and I will rename it to list files.ts. So let's check. First things first, I will modify the interface, which is now called list files tool options. It will accept the project ID and the internal key. It's not going to have any params since it's just going to read all the files that we have. So let's go ahead and change this to, it's no longer called create read files tool, it is create list files tool which uses list file tool options and extracts project ID and the internal key. The create tool itself will have a name list files and the description is also as always important and you might want to copy this one from the source code as well simply because it's long. I mean, or you can just pause the screen and write it out. List all files and folders in the project, return names, IDs, types, and parent.\nID for each of them. Items with parent ID now are at root level. Use the parent ID to understand the folder structure. Items with the same parent ID are in the same folder. So I'm just giving it a little bit of context of what it will receive, right? Now the parameters are just going to be an empty object, so write them as such. To skip the params, you can just write an underscore here. now we're not going to have anything to parse so we can remove all of that now inside of the try method here let's go ahead and do a similar thing so return await tool.step this will be called list files and it's going to be a little bit simpler so I'm going to remove everything inside of here can I do that? I think I went a bit too much okay and we still okay let me just bring this back i think i just needed to remove this there we go all right so tool.step list files the first thing we're going to do is call convex.query api system get project files so we developed this specifically for list files tool and all it does is list all the files in the project using the internal key and the project ID. So we make sure that we pass the internal key and the project ID here, perfect. Now let's go ahead and sort folders first, then files alphabetically. So we've already done this before for our file explorer because it displays it in the exact same file. We're just giving the agent the exact same view now. So we're sorting the files by type first, and then using a simple locale compare for the name alphabetical. And then let's go ahead and structure the file list. So using those sorted files, let's go ahead and create an array of objects with an ID of the file, name of the file, type, and a parent ID or null. Just make sure it's not undefined. So parent ID or null, like this. And then let's go ahead and let's return JSON stringify file list. The error will be error listing files or unknown error. There we go. Another tool finished. Let me remove the extra space here. Let's go back instead of processing message here. And let's do create list files tool. So from list files. and let's go ahead and let's pass in the project ID and the internal key. Now I will order this one first simply because it's going to be used more often than the read files tool. And of course I've messed up my imports so let me just go ahead and quickly fix that. So let's see. Is it still? Okay. So in here internal key and project ID and I change the order that's it so I think that at this point we might already have something so actually I thought we can already try it but we can't try it simply because the coding agent is unused so we defined the tools great but there's nowhere to try it out so okay I'm going to stop at these two tools simply because I think it's better for us to start seeing some results before we write all the tools because we just won't be able to see the results until we finish the entire thing. So how do we call this coding agent? Well, what we have to do is we have to create a network. Again, this is a term from agent kit. So I highly recommend to read about networks to understand how they work and why we need them. Specifically, we need them because we need to create loops. We need the agent to iterate with itself, with its state, and call various tools until it determines that what? There is no more work to be done. That's why you need networks, to create loops, right? Because agents, smart agents work in loops until they're finished. So in order to do that, we're using the create network. so let's import create network from ingest agent kit in the create network give it a name polaris network let's give it agents which is coding agent just a single one and then we have something called max iterations so i keep this at 20 simply because i find it to be a sweet spot basically iterations are how many loops you will allow it to create before you abruptly stop it. This depends on your budget most of the time, right? So if you have an infinite budget, you might run this infinitely, but you most likely want to keep it at something like 20. For example, each tool execution is an iteration, right? So if you have a thousand tools, well, I mean, it's probably not going to use all tools all the time. But depending on how smart your agent is, you might need more or less iterations. Again, this is something you can read more and understand by reading the documentation here. So routing and maximum iterations. You can see why and how you can define what it is. So specifying a max iteration option is useful when using a default routing engine or a hybrid router to avoid infinite loops. Yes, I mean, it's not like it's going to run always infinitely, but there is a chance it can get stuck and it can cause you a very big bill. Great. Now let's go ahead and define the actual router here. So we extract network from here. And what we have to do is we have to get the last result. The last result is network.state.results at minus one. now let's go ahead and check if we have a text response from that last result so last result question mark dot output some message has a type of text and it's coming from an assistant similarly to how we checked if the title is finished right we checked if the assistant message is text so we are now doing the same here we are checking if it's finished so just as we developed has text response let's do has tool calls it's the same thing last result output sum message the type is tool call this should be type safe i believe yes tool call so now it's it depends which model you're using again for example anthropic can output both text and the tool calls together gemini for example doesn't do that or OpenAI. So what I'm going to do here is only stop if there is text without tool calls, final response. So if we detect that an assistant just sent us a text response and assistant has no more tool calls, we return undefined, which basically breaks the router or it signifies that it's finished. Otherwise, we return coding agent, which symbolizes run another loop, basically. So basically, this is the code which decides, should we do another iteration or should we break it? We're finished. All right. Now, let's actually run the agent like this. Result await network.run with a message. now let's go ahead and let's extract the assistant's last text response from the last agent result so again last result result.state results at minus one then let's find the text message if message.type is text and message.role is assistant let's go ahead and give it a default assistant response. I processed your request. Let me know if you need anything else. So we're going to use this if we are just unable to find the last message of the assistant, right? Depending on which AI model you use, this code might work or maybe it expects something completely different. This AI models change very frequently, but you will be able to debug this yourself. If you've come this far into the tutorial, I will show you how you can see the output and then you will be able to tweak things, maybe even ask AI to help you with it, right? So we now have to check if we can modify the default assistant response. So if the type of text message content is actually a string, we're going to use text message dot content. Now on an off chance, it's not a string, but instead it's an array. We're going to map over the content, return the text part of the content, and then join it like so. There we go. And now that we have the assistant response, we can actually update the assistant message. So I'm going to have a little comment here. Update the assistant message with the response. This also sets the status to completed. So update assistant message. And this time the content will be assistant response. There we go. and what I like to do at the very end here is simply return success true message ID and conversation ID so it's easier to debug if it goes wrong alright I think that now we might be able to try this out so I'm just going to copy this entire file here I'm going to create a new file functions.ts and I will paste it inside So I believe this is now saved. It is, perfect. So I will start a new conversation and I will focus on the ingest server too. And let's see. So what files do I have in this project? Did we do this correctly? Did we forget to plug something in? Let's just see. So right now it's generating the title, list project files. And now you can see it is creating the network. It is listing files successfully, so that's great. You can see it managed to call the tool list files. You can see it didn even need the read files tool It just needed a list of files which gave it back an array with functions a type of file parent ID null And I think the response should be, there we go. You have two files in this project, both at the root level. Functions.ts, a TypeScript file. Hello.jsx, a React JSX component file. There are no folders in the project currently. both files are located at the root directory. Amazing. So we successfully made an agent use a tool called create list files and give us a response. Now, I've succeeded with this with Anthropic. I'm not sure what the results are going to be with Gemini. Technically, it should work just as fine with Gemini, but you saw me have a bunch of problems with Gemini before. I keep running into timeouts and limited requests. I'm not sure if I'm doing something wrong. But because of that, I just have to continue developing with reliable models. Otherwise, it's just very, very difficult to get a proper reliable result for the tutorial. Still, if you are using Gemini, let me know in the comments if it works, if it doesn't work, and how you debug it if you encountered any problems. The code itself should work just fine for any model. This isn't tailored to Anthropic, right? The only thing I have tailored to Anthropic is making sure right here that we make sure there are no tool calls because Anthropic also responds with tool calls. For other models, you can just check if there's a text response. It's most likely the last response. With Anthropic, you can't really be as reliable. All right. All that's left to do now is just create more tools. So we only have two tools now and we have to add update file, create file, create folder, rename file, delete files, and also scrape URLs for FireCrawl. I'm going to start with creating the update file tool so we finally see some changes in action. I will copy read files. I will paste it and I will rename it to update file.ts. Then I'm going to update the interface here to be update file tool options with internal key. I'm going to modify the params schema to accept an individual file ID as well as the new content we attempt to update. I'm going to update the entire function here to be called create update file tool, accept this new params, and then return create tool. I'm going to change the name here to be update file. I will change the description to be update the content of an existing file. And I'm going to modify the parameters to match the schema. File ID with a describe of the ID of the file to update. Content, the new content for the file. We're going to start by parsing. So this can stay the same in fact. And the only thing we're going to extract from the parsed data will be an individual file ID and the content. So previously it was IDs, but now it's file ID and content. So now what I'm going to do is I'm just going to validate if the file exists before I even run this step using convex.query API system get file by ID. I've mentioned it will be used by read files tool but it can be used by various tools basically by agent tool right. GetFileById accepts the internal key and the file ID. And in case the file doesn't exist, I'm just going to go ahead and return an early error and instruct the agent to use list files to get valid file IDs. And I will also check if the file type exists, but its type is folder. And I will give the agent similar instructions. File is a folder, not a file. You can only update file contents. Great. So after we do this early checks, which kind of improves the user experience and they don't have to, because if you don't do this early errors, it will get stuck in trying to repeat the API call because it thinks that maybe the network's timed out or something. Basically, the agent doesn't know what it did wrong, right? So that's why you kind of have to do early returns with descriptive errors to let it know, hey, this is not a file ID or this is the wrong file ID. You're probably not looking for that. So the tool step is called update file. And in here, well, it's actually quite simple. Let me go ahead and remove everything in here. The update file will simply call a complex mutation, API system update file with internal key, file ID, and the new content. And let's also make sure the agent knows it was successfully updated. In the error, I'm going to change this to error updating file, unknown error. There we go. That's it. Now I'm going to go ahead back inside of processing message, and I'm going to pass in create update file tool with the internal key. Make sure to import it like so. So let's see if this will work now. Update functions TS to be a simple hello world console log. Nothing else. We're going to see. Does it have enough tools? Does it need any more tools? Also keep in mind, just because it succeeds today, it might fail tomorrow. AI models are non-deterministic. They are unpredictable, right? Sometimes it might work a thousand times and then fail a thousand first time for no reason at all. It's just got confused. Sometimes it's context. Sometimes it's something else. All right, let's see. What did it do here? I've updated functions.ts. It did. Perfect. So the reason it didn't immediately update here is because of the way we defined this component right here. It cannot immediately receive updates because it will reset your cursor position and then it's very annoying to type code. So just, you know, kind of change files or change code something or restart or refresh so you can see the update here. There we go. It works. There are some bugs in styles here. We're also going to take care of that, don't worry. But the more important thing, it works. We can now ask an agent to update a specific file for us. It can find that file and it can update it. And now we're just going to continue, you know, creating tools for other things. Now I'm going to copy update file and I will call this one create files so a bulk update and again if you don't want to write all of this one by one you can just visit the source code of course but I will try my best to at least kind of explain what's going on here even though I think at this point you understand what's going on right so create files tool options accepts project id and the internal key params are parent id an array of files and each of those files has a name and a content. Content can be empty, right? We don't require that, but at least one file needs to be created if we're using this tool. Then we're going to change the name and the props of this tool to create create files tool. This error, I believe, is just a TypeScript server error. If I restart my VS Code, it goes away. So we use project ID and internal key here. I'm going to go ahead and change the name of the tool to be create files. I'm going to go ahead and give it a bit of a longer description, simply so it works better. Create multiple files at once in the same folder. Use this to batch create files that share the same parent folder. more efficient than creating files one by one. So the reason I developed this tool in the first place is because it's very easy to hit maximum iterations if you use just, you know, create file one by one. So this bulk create tool actually helps a lot. Now the parameters are going to be a bit different, more of them. Oops, let me just go ahead and indent this properly. There we go. So parameters are going to be an object. The first item in the object is going to be the parent ID, which is a type of string. And it's the ID of the parent folder. Use empty string for root level. Must be a valid folder ID from list files. So we indicate to the agent. Again, if you forget how to get the ID, just list all files. The second argument in the parameters is an array of files. And inside of that array, we have an object. The object needs to have a name, the file name, which is going to be created, including the extension, and the content, the actual file content. And we describe this as an array of files to create. Right, so exactly what we defined here, just with describe methods. So in the handler, again, we do the parsing, which is fine, but we don't really expect file ID or content. instead we expect parent ID and files now what we're going to do is I'm going to clear up this code here simply because let me see I'm going to clear this up like so and I'm going to clear up everything in the try method we're going to start by calling the tool so return ok like this return await tool step dot run create files asynchronous function let's go ahead and first resolve the parent id now let's check if there is a parent id and if the parent id is not an empty string we going to open another try method here so make sure it has an equivalent catch method down here Let go ahead and do resolved parent ID to be parent ID as ID files Then we're going to use convex query and API system get file by ID to get the parent folder. So we pass in the internal key and file ID, resolved parent ID. and we're basically doing this again to throw early errors so if there is no parent folder we're going to do parent folder with id parent id is not found use list files to get valid folder ids again we're just doing this for early return methods so uh it it under it will be much faster if we throw an early error than if it has to figure out what it did wrong so only if the agent decided to pass parent ID because the agent is allowed to not pass a parent ID if it wants to create files in the root folder. So if it passes a parent ID and if we detect that the parent folder cannot be loaded using that ID, we simply tell it right away, hey, something's wrong. You cannot do this. So if parent folder doesn't exist, we throw an error. If parent folder type is not a folder, we throw an error. The parent ID is a file, not a folder. Use a folder ID as a parent ID. We're just making sure it can understand what it's doing wrong. And in here, we just throw like an overall error, invalid parent ID. Use list files to get valid folder IDs or use empty string for root level. Great. And now let's go ahead and simply do convex mutation. API system create files, passing the internal key, project ID, parent ID, and files. Now in here, let's see which files we have successfully created. Because remember, create files will keep track of all the results which have failed, such as file already exists, and all the files which have succeeded. So we are filtering out all the files which have been successfully created and those which have failed. So be mindful of the exclamation point here. And let's start with a response. You need to give the agent a response so it knows if it did good or not. So created, for example, five files. And then let's go ahead and do if created.length is larger than zero. Inside of the response, let's go ahead and actually display which files were just created. And then we're going to simply do the same if some files have failed, like so. So let me just quickly zoom out so you can see how this looks in one line because when it collapses, it's kind of hard to understand. So you can pause the screen and copy it. All right. So we're just doing this so we have a nice response for the AI model. There we go. And in here, let's do error creating files like this. There we go. Another tool finished. We're going to try this one later. I just want to work on knocking out all the tools that we need. So this is the create create files tool. So import it. There we go. The next one we're going to do is the create create folder tool, which will accept project ID and the internal key. So we don't yet have it. So we're going to go ahead and develop it right now. So I am going to copy create files, paste it here. Rename it to create folder. I'm going to go ahead and change the interface. There we go. It accepts the exact same props. The params are going to be different though. It will accept the name of the folder and the parent ID in case it's a subfolder. I'm going to change the props here and the name of the function to create create folder tool and the same with the props here. Then I'm going to go ahead and change the name of the tool to be create folder. I'm going to go ahead and change the description to be create a new folder in the project. I'm going to change the parameters to be an object. First item in the object will be the name, the name of the folder to create. And the second is going to be the parent ID. So the ID, not the name of the parent folder from list files. Or empty string if we are creating the folder at the root level. Great. So parsing works exactly the same, except we are not expecting parent ID or files. So let's just do name and parent ID. Now let's go ahead and again call await tool step within try and catch, except this one will be here in the create folder. So let's start by validating the parent ID if it is provided. So let's check. If we have parent ID, let's try and get the parent folder using API system get file by ID. And parent ID here. Let's just do as ID files. If there is no parent folder, let's go ahead and throw the exact same error. So parent folder with ID, parent ID is not found. Use list files to get the correct ID. You can, of course, I mean, you can use the same errors. And we do the same thing if the parent folder is not a type of folder. basically as much as possible to let the agent know what it did wrong. Alright. So now that we have the parent ID, let's go ahead and simply call the mutation. So new folder ID, await convex mutation API system, create folder. There we go. Pass in the internal key, project ID, name. And then if we have a parent ID, pass it as parent ID and cast it as ID files or pass along undefined. You can remove the created and the response and we're very simply going to return a static response. Folder created with ID folder ID. Error creating folder like this. Let's go back inside of process message and let's go ahead and import create create folder tool like so. All right, at this point, let's just try it out a bit. So we have two that we have to check. Can it create multiple files and can it create a folder? Create two files named foo.tsx and bar.tsx. Put simple content inside. So let's see if it is able to use the create files tool. That's the first thing we're going to do. So right now it's generating the agent and it's generating the network and it uses the create files tool. And you can see the output. This is what we were doing. Created two files, foo and bar. There we go. Foo and bar and very simple props inside. Now I'm going to say create a simple folder called source. Nothing more. So I'm trying to give it very simple explicit instructions. Right now it should use the create folder function. And it is folder created with ID successfully. And I have the source folder inside. Amazing. Both of our tools are working just fine. So now let's go ahead and add Rename File Tool. The Rename File Tool is quite similar to Update File. So close others. Okay. I'm going to copy Update File, and I'm going to rename it to Rename File Tool. Now I'm going to go ahead and change the interface. The prop is exactly the same. The params are quite similar, except instead of content, it's going to be the new name. Now I'm going to update the props and the name of the tool itself to create the rename file tool and rename file tool options. I'm going to update the name of the tool and the description of the tool as well. Then I'm going to update the parameters to match new name. Perfect. Now in here, instead of extracting content, we will extract new name. And this can actually be the same. So we are validating if the file exists before running the step. If it doesn't, we throw an error. If the type is folder, well, in this case, no, we should allow read file, I mean, rename file to rename folders. That's perfectly fine. So remove that check. And here, let's go ahead and make sure to call this tool step run to be rename file. And we are simply calling rename file. And we are passing along new name. That's it. And this will be, let's go ahead and give the agent as much information as possible. So this will be renamed file name to new name. Like so. And in the error, it will be error renaming file. Perfect. Let's go inside of process message. And let's add create rename file tool. And pass in the internal key. Great. So what else do we have to do? We have to implement delete files tool. So I think out of all of these, the most similar one might be create files. So I going to copy it and rename it to delete files So let me go ahead and start with this Interface delete files tool options Let's change the params to be file IDs. It's going to be an array of strings, like so. As always, I'm going to update the tool name and the props. I'm going to go ahead and change the name. of the tool to be delete files. I'm going to go ahead and change the description. Delete files or folders from the project. If deleting a folder, all contents will be deleted recursively. And then I'm going to repeat the parameters once more with the proper describe handlers. there we go now inside of handler we parse as we usually do and we extract file IDs from here so now what I want to do is I want to validate if all files actually exist before we run this step so files to delete are going to be an arrays and And in each of that object inside of the array, we expect an ID, name, and a type. So let's go ahead and very simply run a for loop. So for each file ID, inside of the parsed file IDs, I'm going to go ahead and attempt to fetch the file using convex query API system get file by ID, passing the internal key and passing the file ID itself. Then I'm going to go ahead and check. If there is no file, let's return an error. File with ID, file ID not found. Use list files to get valid file IDs. So the agent doesn't attempt to delete files which don't even exist. Otherwise let's go ahead and push them to the array. Files to delete, push the ID of the file, the name and the type of the file. There we go. Now we are ready to open try catch and call a tool. So the tool is going to be called delete files. So let's go ahead and change this. Let's go ahead and let's modify everything inside. I think it's easier this way. So instead of delete files, let's start with results, which is an array of strings. We're going to open another for loop iterating over our array files to delete, which we have confirmed exist. We're then going to go ahead and call convexMutationAPISystemDeleteFile for each of those files along with the internal key. And then in the results, we're simply going to go ahead and push deleted file or folder and then the name of that file or folder successfully. And then finally, what we're going to do is return results.join. So to the agent, we're just going to return a plain string saying these are the files we have deleted. Error deleting files. So the agent doesn't get confused. All right. Now let's get inside of process message. Let's add create delete files tool and the internal key inside. Great. So I believe there is one more tool to create. And that is the ability to scrape URLs using file crawl. So I'm going to go ahead inside of tools here. And this time I'm just going to create a new one from scratch. So scrape URLs.ts. Let's go ahead and import zod and create tool. Let's go ahead and import params schema, which will basically just accept an array of URLs. So array of URLs, like so. let's go ahead and export create script url tool let's return the actual create tool util i'm going to invent this back let's go ahead and give this a name of the tool to be scrape urls and then i'm going to go ahead and write a description scrape content from urls to get documentation or reference material. Use this when the user provides URLs or references external documentation. Returns markdown content from the script pages. Perfect. Now let's go ahead and actually add the parameters. There we go. And I believe the last thing we need is a handler. so we get rid of those errors there we go so parameters are just an array of strings there we go okay so inside of the handler itself as usual we get the params and we destructure the step and we rename it or alias it to tool step we do the usual parsing here parse parse schema save parse we break if there's any error with parsing and now from the parse the data, we can extract the URLs. Let's go ahead and open try and catch. Like so. In the try method, we're going to go ahead and open the actual tool called scrape URLs. Let's go ahead and prepare the results. So it's going to be an array of URLs and the content behind the URLs. Let's go ahead and do for const URL of URLs, open another try and catch. In this inner try, let's go ahead and use result from await firecrawl.scrape, pass in the URL and formats into Markdown. Now that we have the result, let's simply check if it's valid and then push it to the array. So if result.markdown is received, simply push it and mark its content as result.markdown. In the catch, we are going to push that we failed to do something. So if whatever error happens, just fail to scrape that URL. Perfect. Now let's go ahead and do if results.length is zero, no content could be scraped from the provided URLs. Otherwise, return JSON stringify results. And then in the catch method here, let's simply return error scraping URLs. If error is instance of error, display the error message. Otherwise, fall back to unknown error string. That's it. let's go inside of process message and let's add the last tool which is create scrape urls tool perfect so i believe our agent is quite capable at the moment so if i go ahead for example and start a completely new project now let me go ahead and do that and if i tell it to for example create a react plus vit app and a simple to do app inside i think it should be able to do that again ais are non-deterministic you can get as many successful results as you can get bad ones right but the more tools you give it the more descriptive you are the better that these tools get and the more tokens and the higher budget you have, obviously the better results you are going to have. So what we've developed right now is a good harness, right? And you can always improve this harness. I'm kind of limiting myself so we don't, this tutorial can go on forever, right? I could be doing this for months, but I have to call it quits somewhere. So that's what I'm doing right now. If you want to create more tools, if you see any, you know, gaps for optimization, go ahead, do it. And you can see how this is happening right now, right? Our agent is just calling a bunch of tools, create files, create folder. Here it is. You can see how it's creating all these files in real time. I suggest like opening these folders so you can actually see when the new files are created. So what I'm going to do now is I'm just going to pause and, you know, Maybe it's going to be successful. Maybe it's not. Sometimes it will be successful the next time you do it. And these things you see, like 1, 2, 3, 4, 5, those are iterations. So if you see it's getting close to 20 or it often gets interrupted near 20, you might have to increase your amount. And here we go. In my example, it was successful. It's telling me how to install it, how to run it, and where to see it. Of course, we can't really do that right now simply because we don't have the preview ready, but that's what the preview is going to be for. It's going to be able to actually install, run, and open the browser. But looking at the code, I see no reason why this shouldn't be working just fine. So yeah, very, very impressive clone of Cursor already. I mean, we are basically finished. At this point, we have finished the majority of things that a very basic Cursor clone would have. Cursor, for example, doesn't even have preview. it doesn't even have export so those are the things we're going to do next we're going to do export we're going to do preview and i'm going to make sure that this markdown looks a bit better because right now the contrast behind this messages is kind of invisible and we've saw some markdowns where it just looks bad so that's going to be a super simple styling fix but i'm going to do that later amazing so that marks the end of this chapter uh we implemented proper message cancellation which you can see now is very useful right if i go ahead and start a new project now and if i tell you know create a react plus vt app yeah and a simple weather app and my agent starts going wild you know it starts calling all of these tokens blah blah blah previously we have no way of stopping that now we can just press stop that's it request cancel you can see that it's canceled it's no longer spending any tokens so that's why that cancellation was important for us to develop initially. Perfect. So let's go ahead and merge all of this. So I'm going to shut down a bunch of my apps right here. And let's see chapter 13. So this will be Git. Well, we can check out first. Yeah. Git checkout new branch 13, AI agent and tools, Git\nAdd dot git commit. It's going to be 13 AI agent and tools. And then git push u origin 13 AI agent tools like so. Once this has been pushed, I'm going to go ahead and go inside of my repository here. I'm going to open a new pull request. and then I'm going to go ahead and review it. And here we have the summary. New features. We added ability to cancel in-progress message processing. We added past conversations history dialogue with searchable list. We introduced comprehensive file management tools, create, update, delete, rename files and folders. We added URL content scraping capability using Firecrawl and our scrape URLs tool. Implementing dynamic conversation title generation. We enhanced message status tracking with processing completion and cancellation states. And now we have eight comments, but they're nothing scary, don't worry. So the first thing here is that we are returning an invalid request on the body here. It says we are, so, okay, request schema parse throws on validation failure and will surface as 500 oh so i should be using safe parse instead of parse all right i see yes because technically this failing shouldn't be an internal server error 500 it should be just a user facing error 400 user sends something incorrectly all right yes and now for the rest of the comments it is simply telling me that i need to handle tool step to avoid returning undefined. So I'm not exactly sure what it means here, but yeah, I think it's just the fact that when I invoke tool step dot run, I use a question mark here. I'm yet to see if this behavior is needed. I think from the agent kit, it's not. I will make sure to research the documentation a bit more, but I do not think this is needed. I think it's fine like this. And all the other comments are referring to the same tool step, just so it doesn't cause a failure. But all of those are labeled as minor. So we did a very, very good job. Almost 3,000 lines changed and 19 files changed. Some of them are, of course, generated files and package locks and package JSON. But overall, amazing, amazing job. Let's go ahead and merge this pull request. And then let's go ahead and let's go back here. Git checkout main and git pull origin main. I almost forgot how to write the command. There we go. Now we are officially up to date on our main branch here. So if I go ahead now in here in graph, you can see that I have detached for 13 AI agent and tools, and then I have merged that pull request back inside. So yes, as I said, I have these two commits here, which I just used to update the readme because I finished part one of this tutorial. You don't have these two commits. That's perfectly fine. Your last commit should be 12, conversation system, and we now finished 13 AI agent and tools. So that marks the end of this chapter. We implemented message cancellation flow. We built conversation history dialogue. We configured AI agent with system prompt, and we created a complete tool execution system. Amazing, amazing job, and see you in the next chapter. In this chapter, we're going to add a preview to our project. We're going to be doing this using web containers and terminal. In order to implement web containers, we have to configure them using a specific package and by setting up specific course settings. We're also going to have to build a complete file tree mounting system because our database stores our files in one structure, whereas web container expects a whole different structure. We're going to implement the actual terminal using a package called xterm.js. And finally, we're going to create preview settings, allowing the users to add custom commands on how to run their project. Let's get started by installing the dependencies we need. npm install at webcontainer forward slash API. Then the second package is xterm, xterm. And finally, xterm forward slash add-on fit. So make sure you have these three packages installed. Next thing we're going to do is set up the foundation on which everything else will depend on. And we're going to start by configuring cross-origin isolation headers. So how do I know that I have to do this? Well, very simply by following the web containers documentation. In here, you can see that in order to configure the headers, I mean, in order to make web containers work, we have to configure the headers, right? But they give us a couple of options. We can either use require corp or we can use credentialless. I have not researched this too much, so I'm not really that familiar with course and all of the headers. But what I do know is that require corp will actually cause problems with some other features that we have in our app, such as billing, which will come in later. Because of that, we're going to be using credentialless. I just wanted to make that clear. So the only reason I'm choosing credentialless is because this enables web containers to work within our project and it also doesn't break anything else in our app. So that's the only reason and I just don't want you to think that I'm pulling this information out of nowhere. There actually is a reason why I'm choosing one over the other. So let's get started by going inside of next.config.ts but of course just before that I always love to show you my package JSON so you can see my packages. WebContainer API version 1.6.1, xterm add-on fit 11.0, xterm, xterm 6.0.0. So let's go inside of next.config.ts. So far, we only have an empty nextconfig and sentry configuration if you have set it up. If you didn't, then you don't have this. But what we need right now is this part right here. So in here, we have to add specific headers. Basically, we have to add these two to our next config. Now, the way you do that inside of next config is by using asynchronous headers. Inside of here, you have to return an array of objects. So let's go ahead and return an array of objects. Let's go ahead and select the following source, basically every path in our project. And then we have to add an array of headers. The headers will be objects, basically defining these two inside. So the first one will have a key of cross origin embedder policy, and the second one will be the value, which is credentialless. And then we're going to go ahead and do the other one, which is cross origin opener policy. So make sure you have that key and value same origin. So basically we have transferred the rule that we need right here into proper next headers. To check if you've done it correctly, you can go ahead and npm run dev your app and you shouldn't see any errors in your app. You should just be able to run localhost 3000 normally with no errors. Great. now that we've got that ready let's go ahead and update our convex schema so you know we need to update our convex schema because of this to create preview settings with custom commands to allow users to clearly define how to run their app because your app can be running on port 3000 3005 5000 depending if you're using create react app or vt or nuxt or next js or a bunch of other apps, right? So that's why we need to allow our users to specify themselves if the AI cannot figure out how to run the app itself. So I'm going to go ahead and find the projects table. Here it is. And after export repo URL, I'm going to extend it with another field called settings. So settings is going to be completely optional and it's going to be an object. The object will require, I mean, it won't require anything. Basically, everything here is optional. But you will be able to pass install command and dev command. Basically, we're going to instruct the web container on how to install the packages in the project. And we're going to instruct web container on how to run the developer script. So we can actually see it. So now that we've added this, let's go ahead and save, and let's make sure that we have npx convex dev running. This will synchronize our new schema, and since this is a completely optional new field, we won't have to delete any of the previous projects. Great. Now let's go ahead and implement a mutation that will allow users to update the settings. So head inside of convex projects.ds, and in here, I'm going to go ahead and I will prepare the update settings mutation. so let me go ahead and close this there we go the update settings is a mutation which accepts id and the actual settings object for the arguments just make sure that the install command and dev command aren't accidentally misspelled they need to match exactly the settings object above great now first things first let's go ahead and verify our identity so we've already done this a couple of times so we can just copy it from here like so. Now once we have the identity let's go ahead and get the project. We can get the project using arguments.id since the id in the arguments is referring to the project. If we don't have a project let's go ahead and throw an error. Project not found. And we also have to check if we actually own this project. So if project owner ID is not identical to identity subject, we are not authorized to update this project We shouldn allow anyone else to change the developer and install script of this project And finally let patch the project stable using the project ID and the new settings we have And let's also tweak the updated ad since it's just been renewed. Perfect. The next thing we have to do is we have to build the file tree utility. So that's referring to this part. Basically, right now, let's take a look at how in our schema, here it is, it's open. We store our files. So here they are, right, files. So when you load files for a project, you basically get an array of objects. And each object has a project ID, parent ID, name, type, content, storage ID, and updated ad, right? so it's just a very flat array this is how it looks like basically an array and then you have an id 1 2 3 type is probably file name is foo.js and then inside you have what is it content which can be something like console.log hello world right and so on and so on with a bunch of objects. And even folders aren't really nested, right? You just have parent ID, one, two, three. So for example, this file would be within this one. Makes no sense right now because then this has to be a folder and it shouldn't have any content, but I think you get the point. Basically, we have a very flat structure which works for us and for our file tree. That's why we've developed it this way. But it's not exactly compatible with web containers, right? Because if you go ahead inside of the web containers documentation here, you will find that their file system expects this, right? So they have a very different kind of structure here. You can see how they do folders, right? We basically have to create a util, which will convert our flat array structure into this. And I'm going to show you how this function looks like. And honestly, I think it might be better if you just went to my source code and copied this function. And you will see why. We're going to attempt to build it together, but it's very, very complicated. I think it might be of use of you to just visit the source code anyway, simply because it's a very complex function. So we're going to go ahead inside of features. And in here, I'm going to create a new folder called preview. And inside of preview, I'm going to create a new folder called utils. And finally, inside file-tree.ts. And the first thing I'm going to do is I'm going to import from my new web containers API the type that they expect, which is the file system tree. And the reason we are importing this is because that's going to guide us into confirming that we've developed the proper function which matches this cast at the end. So let's import document and ID from convex-generated data model. Let's go ahead and define our file document by using document files. So now you can see exactly how our file looks like. And using this type safety, what we're going to do now is very simply convert flat convex files into nested file system tree for web containers. So we're now converting to this right here. and it's basically just a very boring recursive function you're going to see. So let's export const build tree file and we accept files. Now those files are a type of array of file document and what we expect to return from this function is the file system tree which we have imported above. That's the goal. Right now we have an error because we are obviously not returning that. So we have to start by defining a tree, which is an object with a type of file system tree. Then, in order to remove any duplicates and for easier manipulation of these files, let's go ahead and put them instead of a map. So files map, new map, and inside for each file, go ahead and return an array of file.id and then the rest of the file content. Now the first thing we're going to develop is getPath function. so get path function will accept a file which is a file doc type and it will return an array of strings this will allow us to traverse through their parent id to get the full path of the project right so something like components i don't know navbar icon.dsx right that's what we kind of plan on returning here. So we have to start by getting the file name. So initially, the parts of the path will start with just the file name, because that's the one thing we know, right? For example, navbar.tsx. This is what we know about every file. So that's the beginning of the array. And we also know the parent ID. So let's assign file.parent ID here as well. And now we're simply going to traverse up the path. So while parent ID is available, let's go ahead and get the parent using files map dot get parent ID. So this is why putting them in a map is useful because we can very easily just get the exact parent of the file whose path we are just trying to find. In case there is no parent, we can just break the method. Otherwise, let's go ahead and unshift into our array parent dot name. And then let's assign the parent ID to be its parent dot parent ID. And finally, return parts. So basically, in the first iteration of this loop, we're going to go ahead and start with something like icon dot t s x, then this icon dot t s x file will have a parent. And then in the second iteration we might have something like components forward slash icon.tsx if components has a parent we might have something like source components icon.tsx that's what we're doing right now so we have so we are trying to generate a path similar like that all right now what we have to is for each file of files which we have in the parameter we've passed here. We have to generate the path parts using our getPath. So passing the file here. Assign the current to be tree which we also have. Let me just find here. So this is the tree in this stage of iteration. So basically we're just going to be adding files to an object until it looks like this. So let current is a tree. Let's go ahead and open a for loop within a for loop. Let iterator be zero. Iterator is smaller than path parts dot length and iterator is increasing. And now in here we're going to go ahead and change the part to be whatever is the current iterator from path parts and we're also going to check if that part is last. So if iterator is equal to total amount of path parts. In case if it's last we also have to check if file.type is folder. Now if it is we have to display that in this type of structure. Now, in order to do that, this is how we do it. We're going to add to the current object for that specific part a very simple indicator of an empty directory, like so. Else if not file.storageId and file.content is not undefined, so So else, this is a case for last file and if it's a folder. This is a case for text files. We are purposely skipping storage ID or binary files because they just increase the complexity of this by 100. We might revisit this later, but right now we're not even working with storage with binary files, so it's fine. So let's just go ahead and make sure that we can add normal content here. So that would be, again, current part, open an object, file contents file dot content because usually for storage id we would have to load the storage id and then load base 64 content inside of it so uh really a lot of complexity for something that we don't even have yet now that's the case for is last if it's not last we're going to go ahead and do something else. So let's check if we don't already have that inside of our current tree. So if we don't have that part, let's go ahead and simply add that part like so with an empty directory. And then let's go ahead and do const node current path. My apologies, not path part. if there is directory in node, current is node.directory. It's very easy to get lost in this function. It's really not a simple one, right? But it is what we have to do. And make sure to return the tree at the end. Okay, so you shouldn't have any errors here. You should now successfully accept flat convex files, and it should return this. my project directory, foo.js file, and then the contents inside, or empty folder, just directory. So that's what we were doing right here, right? Feel free to open the source code for this. You know, this is super complicated. Even I use the AI assistance on this. I get very easily lost in like these kinds of recursive functions. Actually, I don't think this is recursive at all, but I think you get what I'm saying. Okay. Now what we have to do is we have to create a simpler method, which will help us get a full path for a file by traversing the parent chain. Very, very similar to what we did here. Except this doesn't actually create the full path. This just returns an array of strings, right? So what this does is it accepts an input. For example, let me show you. It accepts an object id123 console log like this name foo And it also probably has a parent ID of 3 to 1, right? And it returns an array of something like source, components, foo.js, right? That's what this get path function does. but what we're going to develop now at the bottom is actual source components foo.js in like a breadcrumb type of string so nothing we haven't done already right so let's go ahead and define a function it will be called get file a path and the function itself will accept two params the actual file that we're trying to traverse through and the file is map basically a map of ID of files and file doc. Let's go ahead and very simply define the parts starting with file name exactly as above. Let's go ahead and define the parent ID to be file.parent ID. And again while the parent ID is active let's go ahead and traverse up the chain by getting the current parent. If there is no parent let's break but if there is let's go ahead and add to the array at the start of the array. That's what unshift does at the start of the array. And let's assign the new parent ID to that file's parent ID. And the only difference we're going to do is instead of returning parts, like we did here, is parts.join. So why did I just duplicate the function again? Well, because we are going to use this independently of this one. Basically, we are going to repeat this many times. It's a very useful function. All right. So again, feel free to just open the source code if you think there's a bug here and you can just copy it and then it will work right away. But I kind of tried to explain what we're doing here. Basically, we have our file structure and we need to convert it to this file structure. Great. Now, let's go ahead and implement the web container hook. The web container hook is what will actually start the web container, use these functions which we've just developed, and allow us to, well, see something, right? Do we need a hook for that? Well, no, but since we are in React and XJS, we work with hooks, and I kind of feel like this is a natural way of using web containers. So let's go inside of Features, Preview, and in here I'm going to create hooks. and in here let's add use-webcontainer.ds. Perfect. Let's start with use client. Actually no need to start with use client because we are only going to use this within a client. Instead let's go ahead and start with all the imports. So make sure you have use callback, use effect, use ref and use state from react, use query from convex react and web container from WebContainer API. Now let's go ahead and reference our build file tree and get file path features. And it looks like I have forgotten to export one of these. So let me quickly go back here. Utils file tree. I'm exporting a build tree file and I wanted to name it build file tree. There we go. So that resolves the problem. Okay. Then let's also import the usual suspects, API from generated API from convex and ID from generated data model. What we're building now is a singleton web container instance. So we're going to need to have a let web container instance, which is a type of web container or null, and boot promise, which is a promise of web container or null. Right now they can only be null, so this is throwing errors. But later we will change it to have this other type, so it won't throw any errors anymore. We're going to start by defining a function called getWebContainer. This will return a promise and web container inside. Like so. So first things first. If we already have a web container instance, we're going to return a web container instance. This way, we don't have two instances. Then if we have a boot promise, if we don't have boot promise, my apologies, we're going to start a new boot promise using webcontainer.boot. And what this is referring to is our cross-origin policy. And if you remember, we've set it to credentialless. So instead of next.config.ts, we've set the value to be credentialless. And basically, because of the instructions here in the documentation, let me go ahead and show you, configuring headers, if you switch to credentialless, you also have to boot your web container specifying this key right here to be credentialless. So that's where I got that from. All right. And if you're wondering about exact documentation for this, it's also here, but it will not exactly show you how to do it like I'm doing it. You can see how to create a web container instance using webcontainer.boot. But this is more like an overall guide on how you would quick start it. What we're doing is basically a compilation of a bunch of these references and ways of booting it into a hook. That's kind of the complicated part. But feel free to look through quick start so you can actually see some of these functions that I will be calling here. All right. So, so far, we are making sure that we are not able to boot the web container instance twice. That's why we're checking if there is no boot promise, only then assign a web container boot with credentialless headers. Then let's go ahead and set the web container instance to be await boot promise. And finally, return web container instance. And just like that, we've resolved those two errors right here because now in the runtime we actually assign them to their types. Great so that's it for get web container. Now what we have to do is develop a very simple method to tear down the web container so we don't have any memory leaks. So we're going to tear down the web container if we have the web container instance. So if the web container instance already exists let's do webcontainer instance.teardown. And let's do webcontainer instance and assign it back to null. Let me just fix the indentation here. And outside of the if clause, set the boot promise to be null. There we go. So now we have teardown webcontainer and get webcontainer functions ready. What we have to do now is we have to develop an interface for our hook. So our interface, use webcontainer props, we'll accept project ID, which is a type of ID complex projects, enabled, which is a boolean, and optional settings, which we define in our database. Remember, install command and dev command. If you're unsure, you can always open your schema.ts file and go inside of your project stable and find the settings. Make sure you have install command and dev command, and don't mistype them here. Great. Now we have the interface for our hook, which means we are ready to start building the actual hook. So useWebContainerHook uses the same named props here, accepts project ID enabled and settings. The first thing we're going to do is we're going to set the status of this web container. So status, setStatusFromUseState. It can either be idle, booting, installing, running, or error. And by default, it's going to be idle. and while we are here let's also define all other states which we are going to need starting with the preview url which can be a string or null and by default it's going to be null then let's go ahead and add the error which can again be string or null the restart key will very simply be used to change the key of an element changing the key of an element in react makes it re-render entirely. So we're going to use this as kind of a hack to refresh the web container. So in case it gets stuck or it boots incorrectly, the user can always forcefully restart it. So it installs the dependencies again. Then let's go ahead and also prepare the state for the terminal output, which can be a string. And well, when you define the default type and it can only be that type, you don't have to define it. The same way we didn't have to define number here, right? Only when it can be multiple types does it make sense to define it, like string and null. Great. Now let's go ahead and prepare some references here. So we're going to need two references. The container ref which is useRef and it can be either web container type or null. And by default it's going to be null. and has started ref, which we are very simply going to be used to prevent some duplication. So by default, this one will be false. Great. What we have to do now is we have to fetch files from Convex. And this is where Convex real-time actually comes in so handy. Convex auto-updates on any changes, which means we accidentally developed hot reload just by using Convex. So if I want to get my files, all I have to do is call useQuery from Convex and call API.files.getFiles and pass in the project ID from the files, which I need. Or let me check even further. Maybe I even have a hook for that instead of useConversation. UseFile. Actually, this will be in the projects, I believe. Hooks, useFiles. Do I have that? looks like I don't have used files so let's go ahead and quickly create use files here use files and all it accepts is a project ID and I'm not sure if it should be able to be skipped so dish will be an idea of projects get files either pass in the project ID or skip it. So just like that we developed an abstraction use files So now in here I can just call use files and pass in the project ID There we go. Use files from features, projects, hooks, use files. And I'm going to move it here, and I'm going to remove the import of use query from convex react. I think this will work just fine. We're going to see later if it doesn't by chance. so what we ought to do now is boot the actual web container so initial boot and mount using a use effect let's prepare an empty use effect like so and now in here we're first going to check if we should prevent this from happening so if we are not enabling this or if there are no files or if files.length is zero and remove the exclamation point here or if has started ref.current is true. If any of that happens, we have to return early and not do anything. It either means we didn't enable it, there are no files to load or we have already started and this is an accidental reboot. So we're immediately going to change this to true then. So now it makes sense. If this happens twice, this will prevent that from happening. Now let's go ahead and develop the actual start method, which is going to be an asynchronous function. Let's go ahead and open a very simple try and catch block here. Instead of try, let's go ahead and set the status to be booting. Then set the error to null. then set terminal output to be an empty string. This is kind of like a reset. And now we have to create a function to append the output to terminal. So append output accepts data, which is a string, and it returns set terminal output. And it will simply append the data to the current value of the state, like so. Then let's go ahead and actually get the web container and assign it to a ref. So container is await getWebContainer, a function we've developed first here. And then we simply assign that to a reference. Then we have to build the file tree so we can actually mount the files to a web container. So file tree, buildFileTree, files. And let's go ahead and mount that using await container.mount file tree. You can see that if we didn't build the buildFileTree function, these files from convex which we load here would be completely incompatible right you can see it's an interval incorrect structure that's why we have to build the file tree first then let's go ahead and look for an event called container on server dash ready skip the port only focus on the url and set the preview URL to that URL we've received from an event server ready and change the status of this hook to running. At this moment, we can also set the status outside of this event to installing because this won't go before this one. Only once we receive server ready will it change the running. So ignore the fact that we are defining this before we define this. After we set this to installing, we actually have to parse the install command, which by default is going to be npm install. So how do we define the install command? Well, very simply, we can use the settings. Let me go ahead and find where we define the settings. Just a second. So we pass the settings. We could technically also fetch the settings. That might also work. but then we'd also have to restart the project carefully. So I'm going to use it as a prop for now. So it's going to be settings.install command or npm install, like so. And then go ahead and use install command.split. Split it basically by space, because the way web containers accept install commands are in a very specific way. So what we have to do is basically separate this array, install bin, and then the rest of the install arguments. Like so. So we're basically going to have an array npm and install or npm run install, whatever the user specifies. And then let's go ahead and append output to a terminal. So the terminal shows the user exactly what we're doing right now. So the append output will have install command and a line break. And now we actually have to spawn this command. So all of this right now is just cosmetics. And now we're initializing install process with await container.spawn and passing the install bin command followed by the rest of the arguments inside of the command. Great. Now let's go ahead and create a writable stream from the install process. So install process.output.pipe2, execute that, and call new writable stream. Open an object inside. Define the write function, which has data as a prop, and very simply use the append output and pass in the data here like so and it's not write stream it's writeable stream how do you write this write double stream there we go so I believe that I might have imported write stream from fs so if you have done that you can remove it okay so it's writeable stream okay And this will basically display the entire output of the install process into the terminal. Now let's go ahead and also catch the install exit code using await install process.exit. If install exit code is not zero, it probably means there is an error. So let's go ahead and throw an error. throw new error and inside we can just add a template literal showing the command we attempted to run fail with code and then show the code which was thrown because the exit code can be zero which basically means okay I've successfully finished npm install or it can be something else so we'll just show that back to the user great what we have to do now is we have to parse the developer command the developer command is basically npm run dev something to start the project and we're going to go ahead and use the same logic right so developer command is a parsed through settings dot dev command keep in mind that both settings and dev command can be optional so we have to add a fallback npm run dev so we're going to assume most of the projects will be run with npm install and npm run dev but of course users will be able to define their own and then we also have to split the dev bin and the dev arguments using .split with an empty space. Let's go ahead and immediately append output to the terminal with a new line breaks and the developer command. Let's go ahead and initialize the actual dev process by spawning this. So developer process await container.spawn developer bin and the rest of the developer arguments. And now we just have to do another writable stream here. So developer process.output pipe2 new writable stream, call the write function which accepts the data and simply appends the output to the terminal of that data. So everything that's happening during the spawning of this command will be shown to the user in their terminal, simulating the exact experience you would in a real terminal. All right. And in the catch method here, let's go ahead and do catch error. Set error. if error is instance of error do error dot message otherwise unknown error like so and set the status of the entire hook to error so for example when we throw from the install code that will be called right here and set the status to error perfect and now execute the start method like so so we've just defined the entire start method right but they never called it. So make sure that at the end you actually call it. And now for the dependency array, there are a couple of those we have to add. So let's add enabled. Let's add files. Let's add restart key. Let's add settings, question mark dot install command. Well, dev command, install command, both of them are needed basically. Great. Now let's go ahead and implement a simple hook to enhance the hot reload of the files. So sync files, file changes, hot reload. This will be another use effect, though much simpler. So let's open an empty use effect once again. Let's go ahead and check if we have the container from our ref. If there is no container or if the status is incorrect or if there are no files, basically if any of these cases happen, let's do an early return. What we ought to do then is create a simple file map. So file map, new map, files.map, get the individual file and add them in an array showing the file ID and the file content as the other part in the array. Great. now let's go ahead and open a simple for const file of files let's go ahead and check if the file is an actual type of file and it has content and it's not a binary file we can do that by adding an if clause if file.type is not a file or if file has a storage id or if file has no content at all just continue no need to do anything but if the file is an actual file which has text content Let's go ahead and define the file path using get file path util. Pass in the file and the file is mapped. And then we can go ahead and write that file to the container.\nWrite file, file path, file.content. In a dependency array, add files and status. There we go. Let's go ahead and reset the entire thing if we receive a disabled event. So this is a much simpler use effect. Here it is. So if we reset the enable prop, so if not enabled, immediately change has started ref.current to false, set the status to idle, set preview URL to null, and set error to null. Basically, a complete reset. And then let's go ahead and just implement a function to restart the entire web container process. So we're just doing some teardowns now, right? So restart is going to be a use callback. And the first thing it's going to do is going to call a function teardown web container. After that, it will set the container ref.current to null. It will reset has started ref.current to false. It will set the status to idle. Set the preview URL to null. Set the error to null. And finally, we're going to forcefully increase the key, which will, again, just reset everything even more. All right. And finally, let's return status, preview URL, error, restart, and terminal output. That's it. That's our complete hook. We're now ready to develop the actual terminal component. Our next task is to implement the terminal component. Fun fact, the library which we're using, xterm.js, is actually used in real VS Code and various other projects. So let's go ahead and see how we can implement a terminal component using the package we installed, xterm.js. So I'm going to go inside of features preview, and I'm going to go inside of, let me see, I have to create a new folder called components. and I will create preview-terminal.dsx. I'm going to start by marking it as useClient and then I'm going to import useEffect and useRef from React followed by Xterm packages. Basically the terminal and fit add-on package. The fit add-on package will be used because the terminal will be within an allotment pane which can be resized. So because of that, we need that package. Otherwise, you can develop it without this package. And we also need to import the CSS for the extern. Let's start by creating an interface, previewTerminalProps, which very simply accepts the output. So in order to actually develop the component, we need to export previewTerminal and define the output here. All right. now inside of the preview terminal let's add a few refs we're going to have a container ref which can be html dev element or null we're going to add a terminal ref fit add-on ref and last length ref all of this then let's go ahead and create a use effect which will initialize the terminal. So I'm going to go ahead and open an empty use effect like we usually do. And I will first check if we are ready to run the terminal. So if there is no container ref, or if there is no terminal ref, let's do an early return. Otherwise, let's go ahead and initialize a new terminal. Now inside of these options here, you can add a few settings. Now, I will configure it the way I prefer it and the way I found it looks the best for our project. You can, of course, tweak this later on. So I'm going to enable this setting. I'm going to enable this setting. I will set the font size to 12. And lastly, I'm going to add font family and a background color. So font family will be monospace and theme will use this specific background. now outside of this terminal constant let's go ahead and define a new fit add-on plugin so this fit add-on plugin will very simply be loaded into a terminal using their built-in load add-on function and after that we are ready to open the terminal in the container which we store in the And then let's simply go ahead and add both of those, fit addon and terminal, to their refs. So terminal ref.current gets the terminal and fit addon.ref gets fit addon. We now have to write existing output the moment we mount the terminal. So, if there is any output, use terminal.write and pass in the output, and also change the last length ref.current to be output.length. Then we're going to add a requestAnimationFrame function, callback, and call fitAddOn.fit every time a new animation frame is rendered. This way, we can have a terminal which expands within our resizable panels. Let's also add a resizable observer. So resizeObserver calls new resizeObserver and also calls fitAdon.fit. And we actually have to observe something, so let's observe the containerref.current using the resizeObserver.observe function. The last thing we ought to do is a cleanup function in this hook. so in the cleanup function we're going to disconnect the resize observer we're going to dispose of the terminal and we're going to reset our refs then let's go ahead and add the output into the dependency array right here let me just see do we actually need, well yes output is only used to write existing output. It's not used to be updated, so do not add it here. I'm going to add a little comment here. Output does not need to be a dependency since it is not intended to update anything just used on mount. Intended. All right. then let's go ahead and create another use effect which will be used to write the received output so this is where we will use the output in the dependency array so let's go ahead and define this passing the output in here because now we will need the output so again we're going to check if we have no terminal ref.current or if the output.length is smaller than last length ref.current. So if output.length is smaller than last length ref.current, let's simply clear the terminal and reset this ref we are tracking back to zero. I found this helps with the resizable issue because there was some issues I was kind of fighting the terminal to work within resizable panels. So this is one solution that I found helps to clear up some of the content. Then let's go ahead and define new data to be output.slice lastLengthRef.current. And if we have new data, let's go ahead and write it to the terminal. We can access the terminal instance using our ref. So if we have new data, call terminal ref.current.write and pass in the new data. And update the last length ref.current to be output.length. There we go. One more thing we have to do is a very simple return method. It will return one single div, a self-closing div. And this div will have a reference of container ref. And then it will have a class name. flex 1, minimum height of 0, padding 3. And now we're going to have some very specific styles for the terminal. So we're going to be using this a lot. So feel free to copy that. And the classes I'm adding now is basically just very specific styling of the terminal. So this is one class. Never mind that it's collapsed. See? besides height full we're also going to have x term screen to be height full and background color will be sidebar there we go that is our preview terminal component great now that we have that let's go ahead and implement the settings popover component which will allow us to change the install script and the dev command. So I'm going to go ahead back inside of features, preview, components, and in here I'm going to add preview-settings-popover.tsx. This will mostly be a form. So let's go ahead and mark it as use client, import zod, and let's import everything else we're going to need, which is going to be use state, use mutation, use form, settings icon. Looks like we don't have tan stack react form. So did we not build any forms before? We do have form itself, but we don't have this. Basically, ShadCN added a new way to write forms. So you can now write forms either in the old way. Let me find form. Where is it? I cannot find form. Maybe I should search for form. Oops looks like something not working on the website All right so if you scroll down and find forms you will find React hook form which is I believe how we built forms so far Or no, this is the new one. Okay, so they now have basically, either you can use the React hook form or 10 stack form. And 10 stack form is the new one. So I think it might be better to, you know, teach you how to use this one simply because I don't know I mean I've built the react hook form many times but I didn't build this one too much so I just want to find a way to install this because I can see that I have a missing 10 stack react form and I will save this file anyway and I'm just going to go ahead and research a bit inside of my source components UI. Do I have something called a field? I do have. So I think I should be able to run this normally. I'm just surprised that Shatzian command didn't install 10 stack react form. So I'm going to check in my package JSON to confirm and we truly don't have tanstack react form. So what I'm going to do is I'm just going to install it. I think that's the only package we need. So npm install at tanstack forward slash react form. Usually these kinds of things get installed by running this command but looks like it's somehow missed now. So let me see if I have it now. Here it is and I'm using 1.27.7 version. Alright, so back to business, 10 stack React form. Besides these imports, we're also going to need a button component. We're going to need all the imports from the popover component, which is popover, popover content, and popover trigger. We're going to need to import all the field components, field, field label, and field description from components UI field. And we're going to need to import the input from components UI input. Let's go ahead and import API. And let's go ahead and import document and ID from generated data model. I'm going to start by defining the form schema, which is an object which accepts install command and dev command. We've already seen this a few times. Then let's create an interface preview settings popover props which accepts the project ID for the settings, initial values if we already have some settings, and an on save method. Then let's go ahead and actually define and export a component. So preview settings popover uses the props, accepts project ID, initial values, and on save. It will have its own open and set open use state. It will have a very simple method to update the settings calling useMutationAPIProjects update settings. If you wish to, you can always abstract this. Let me see. So this is for projects. So this would be inside of projects hooks. We have useProjects. So yeah, you could do exportConst. Let's see what we call it useRename project. So this will be use update project settings like so and it will just return this and then later you can add optimistic mutation. Let me see do we have any to do for optimistic mutation. We do not. To do add optimistic mutation if you want to improve it later. So I like to abstract them this way especially because of those optimistic mutation things. it's way easier to maintain that in a different file. So that's what I'm going to do. I'm going to import it like that. I'm going to move it here separately and I'm going to remove the useMutation import since I no longer need it. I can directly access update settings from the hook now. Great. Now what we ought to do is create the form. So we're going to start by calling the useForm hook from tanStackReactForm. So this is a different hook than your usual React hook form. We're going to define the default values, which can be install command and developer command, which come from the initial values prop. If we have it, we use it. Otherwise, we fall back to an empty string. Now for the validators, we're simply going to use onSubmit, validate using form schema, which we defined right here. after the validators, let's call the actual onSubmit method which is going to be asynchronous, accepts a value my apology, yes, just a single value and you can see how in TanStack React form we define all those things instead of use form whereas in the React hook form you have to do it kind of all over the place I think I kind of preferred this one, to be honest now inside of this on submit let's simply define what we do await update settings pass in the id to be project id and the actual settings install command value dot install command or undefined if we ever want to reset the values and developer command to be value dot developer command or undefined. There we go. After that we close the popover and then we call the onSave callback if we have it. There we go. Now let's go ahead and define a very simple const handle open change method which will receive a new isOpen value which is a type of boolean and in here what we're going to do is check if isOpen. If it is call form.reset with an install command of the initial values. Check if we have install command or fall back to an empty string and then do the same for dev command. And finally, outside of the if clause, simply call set open to is open. So every single time handle open change triggers, if we actually open it, we're just going to reset the form. Great. And now we just have to build a composition for the popover. So let's go ahead and return the actual popover element, which will accept an open prop and non-open change, which we've defined just now. Then we're going to have a popover trigger, which will have an as child prop, so it will actually become the element inside. The element inside will be the button with the size of small, variant of ghost, class name height full and rounded none, and the title previous settings. And inside, we're simply going to render a settings icon, which we've imported from lucid react with a size 3. Then it's time to build the popover content, which we composition outside of the popover trigger. It will have a class name of width 80 and then a line of end. Inside, we are working with a normal form element. So a native HTML form element which has an on submit in which we prevent the default and simply handle submit from the form hook. So that form is referring to this form right here. Then let's go ahead and add a div here so we can separate the fields. I'm going to add just one more so we have further separation. An H4, which serves as a label, preview settings. Let's give the heading 4 a class name of font medium and text small. And the paragraph configure how your project runs in the preview. Let me fix the typo. And let's give this a class name, text extra small, and text muted foreground. Then let's go ahead and develop our first field. So that's going to be outside of this div right here. And we're going to add form dot field like so. And let's go ahead and give it a name, which can be install command. And then we're going to render a field by extracting the field prop and using the field composition. So field label, let me go ahead and fix the typo here. Field label will have a text of install command and HTML for field.name. So these are completely accessible fields. And below the field label, let's add an input with an ID of field name, name of field name, value field.state.value on blur field.handle blur on change event field handle change event target value and the placeholder npm install indicating to the user what is a default value and a little description to explain to the user what this is and then we can repeat the entire thing so I'm going to paste out the entire thing here. Here it is. Another form field, this one for dev command. Again, we extract the field, we render the field label with HTML4 field.name, and we simply render start command inside. And then again, another input with the exact same props with a different placeholder, npm run dev, and a slightly different description explaining that this is a command to start the developer server. So these two are identical. This one is for dev command, and the one above is for the install command. Great. And now let's go ahead and add one thing you probably didn't see before, which is a form.subscribe. And inside of here, it can have a selector, and it can look for can submit and is submitting. And then using those fields, we can go ahead and render something inside. And what we are going to render is a button component which will have a type of submit size of small class name with full It will be disabled if we can submit or if we are already submitting And if we are we going to display saving Otherwise we going to display save changes There we go. That's it for the preview settings popover. And that's what it's like to work with Tanstack form. I personally prefer this over React hook form. And I'm going to remove API from here. Not to say that React full form is an amazing library. It absolutely is. I just always feel like developer experience with ThandStack is just a tiny bit more nicer. Great. Now let's go ahead and start bringing everything together and finally rendering this. So we're going to go inside of features, projects, components, and I'm going to create preview-view.tsx. I'm going to go ahead and mark this as use client. I'm going to import use state and allotment. And then I'm going to go ahead and import loader to icon, terminal square icon, alert triangle icon, and refresh CW icon. I'm going to import our use web container from features preview hooks use web container, which we've developed. I'm going to import our preview settings popover from features preview components preview settings popover, which we just finished. Then I'm going to add the preview terminal from the exact same place. We've finished all of these components already. Then I'm going to add a button component. I'm going to add a very simple use project from Hook's use project and I'm going to add ID from generated data model from Convex. I'm going to export const preview view which very simply accepts project ID, which is a type of ID projects. In here, the first thing I'm going to do is I'm going to load the project using the hook. Project, use project, project ID. Then I'm going to decide whether I should show or shouldn't show the terminal using a simple state. Then we can go ahead and call useWebContainer finally. Now useWebContainer needs to have a couple of properties such as the project ID so we know what files to load and what files to transform into a specific file tree, whether it's enabled or not, and finally the settings for the project so the user can change the install command and the developer command. The web container gives us the following items, status, preview URL, error, restart, and terminal output. Let's go ahead and show the loading state if the status of the web container is booting or installing. And now let's go ahead and start rendering the entire thing. so we're going to start by defining a div which will take the full height initiate the flex system and a background then we're going to display a navbar with a height of 35 or as written here 8.75 so i'm going to change it to that flex item center border bottom background sidebar and shrink zero. After that, I'm going to go ahead and add a button allowing us to refresh the web container. This button will have a size of small, a variant of ghost, class name of height full and rounded none. It will be disabled if the web container is loading. And on click, we will call the restart method from the web container. The title will be restart container, and it's going to have a refresh icon. Then what we're going to do is we're going to display something like a little URL bar showing us exactly what URL we are loading. So this is how that's going to look like. A div with a class name flex1, heightfull, flex, itemscenter, px of 3, backgroundcolor, borderx, text extra small, text muted foreground, truncate, and font mono. First thing we're going to check is are we loading. If we are loading, instead of displaying the URL, what we're going to do is simply display a loader to icon from Lucid React. I'm going to give it a class name of size3 and animate spin. Then I'm going to give this parent div a class name flex items center and gap 1.5. And then I will check the status. If the status is booting, I'm going to display a more user-friendly starting, otherwise installing. So the user is aware of what's actually happening. And then I'm going to check. If I have the preview URL, instead of a simple span, I will display that preview URL. I'm just going to make sure to truncate it so it doesn't overflow. So let's give it the class name truncate. Then I will check if not loading and if not preview URL. Preview URL. And if there is no error either, I will just add a span ready to preview. Great. and one more button actually two more buttons we have to add one is to trigger the terminal so a button size small variant ghost class name height full rounded none with the title toggle terminal on click set show terminal and what i'll actually prefer is if we just reverse the current value. This way it will not get conflicted with any other async state. So we just toggle the current state of the terminal, like show it or hide it. And then we need to add a button which will open the preview settings popover, which is very easy because we just have to render the preview settings popover and pass it project ID, the initial values, and on save. That's it. and that automatically renders, if you look down, the trigger, which is a button with the exact same size, variant, and class name as its siblings, so it looks exactly the same. Great. Now, outside of this div, which represents the navbar, we have to render the actual content, right? So, I will start with a class name flex1 and a minimum height of 0 and then we're going to go ahead and start working with allotments. So we start with the main allotment and we're going to make it vertical. Then let's add the first allotment pane inside. This allotment pane will first check if there is an error present. So if we have any error from the web container let's go ahead and display that. that's going to be size full flex item center justify center and text muted foreground inside another div just to send three further flex flex column item center gap two maximum width medium mx auto and text center i'm going to add an alert triangle icon with class name size six I'm going to add a paragraph showing the exact error. And I'm going to display a button component, allowing the user to restart. So on click restart, size small, variant outline, refresh icon, and the restart label. So this is the error state. If something goes wrong with the web container, this is what the user will be shown. Now let's go ahead and display a very similar loading indicator. So, still, within the allotment pane, let's check if we are loading and if we have no error, in that case display a div with size full, flex, item center, justify center, and text muted foreground. Within another div, flex, flex call, item center, gap 2, maximum width medium, mx auto, and text center. Finally, loader 2 icon with size 6 and animate spin, a paragraph with text small, font medium, and the label installing. And the moment you've been waiting for, the actual preview URL. That's the easiest part. Just a very simple iframe. Source, preview URL. This is why we needed to set up proper course. Otherwise, iframe would not be able to be loaded. And the class name, size full, border 0, and the title of preview. Great. The last thing we ought to do is the allotment pane for the terminal. So outside of this allotment pane, go ahead and add a check if we should show the terminal or not. If we should, go ahead and open an allotment pane with a minimum size 100, maximum 500, and preferred size of 200. Then in here, a div, height full, flex, flex column, bg background, and border on top. Within, another div, with a class name, height 7, flex, item center, px3, text extra small, gap 1.5, text muted foreground, border bottom, border border with a 50% opacity, and shrink 0. Then I'm going to add a very simple terminal square icon from Lucid React with class name size 3 and a label terminal. And finally, let's render preview terminal component with an output terminal output, which we receive from useWebContainer hook. And that is it. We are now finally ready to display this inside of the actual page. So let's go ahead inside of, let me just remind myself, this is inside of projects, components, project ID, view. So in here we have two allotment panes, one for the file explorer, one for editor view, but actually this isn't in an allotment pane. This is in kind of like this tab switcher, right? So we have a tab for editor, but we never developed the tab for preview. So finally let find this where active view checks for preview and instead of rendering this let render preview view make sure to import it and pass in the project id project id make sure you have preview view imported so we were just working on this i believe with all these components and we are now ready let's go ahead and check it out so i'm going to go ahead and revisit the last op i have developed uh oh or maybe i have maybe this isn't the one this one basically the one that apparently should be working and it should be a simple to do app so my prompt was create a react plus vit app and a simple to do app inside and when i click on preview you can see that now i am installing it here here and here so i have synchronized outputs everywhere i'm gonna go ahead and try toggling the terminal. You can see that I can do that. I can also click on here and I can see the preview settings with my install command and my start command. So what I'm going to do now is I'm just going to leave it for a few seconds and looks like it's working. There we go. You can see the allotment pane can be moved and now I can go ahead and add hello world, add to do and just like that it works very very good. You can see that even the actual code works. So I'm super super happy with this one. Basically, the only way you can test whether this works right now is with a working code, right? If you go ahead and just start a new project and go into preview, there is nothing to preview. You can try adding something like index.js and try console.log.hello world inside. And this will be synchronized. Sometimes you might need to do a refresh, especially in development. it kind of uses the old one but you can see it fails because it's not a real project it doesn't have a package JSON so you need to be able to at least ask your AI agent create a minimum how do I preview Apple project that can be started with web containers so basically ask your AI to do something like that okay my message failed to send maybe I have some tokens or something oh my ingest is not running npx so you probably have the same error then let me try npx ingest there we go let me start again I will start a completely new project go here and I will again ask it to create a minimum previewable project that can be started with web containers. So yes, you basically have to ask your app to create something simple. And then I'm going to try a few of these examples, and I'm going to kind of purposely try and change the install script so we can see if the preview settings are working, because everything else is working just fine. So I'm going to pause, and we're going to see the result. And here is my result. So just super simple index HTML, a super simple package JSON, which is as npx serve, some readme, script.js, and styles.css. So you can see it should be able to generate something like this. And you can see it works, right? It's successfully run npm install and npm run div. So what I'm going to do now is I'm going to change my dev script to be 4000, for example. I'm purposely going to do that, and I'm going to do a hard refresh here. So now I am expecting this to install, but to fail when it needs to start. Okay, I couldn't do that. Oh, because the port doesn't matter. My apologies. This matters. So let's see. What does it run? It uses npxserv-s. So it runs npm run dev. So if I change this to, I don't know, custom, right? And do a hard refresh. I now expect, again, install to work. But I think, again, it keeps working. I'm not sure why. I'm trying to make it fail. oh looks like I don't know why it keeps working oh this didn't update it seems let me go ahead and make sure this is updated do I have my convex functions ready custom perhaps we have some bug if it's not updated okay now it's custom let's give it a third try there we go missing script dev so now it's failing okay just make sure that your code was actually saved. I probably refreshed too fast. And now I'm going to go ahead instead of project settings, and I'm going to change this to npm run custom. And I'm going to click save changes. And let's see if it will work now. And there we go, because we are able to change to npm run custom. What an amazing job you've done here. You developed the entire preview tab, the URL, the web containers, the terminal output, absolutely everything. One thing I want to make you aware of before we finish the chapter is the license of web containers. So web containers are absolutely the best solution for this. You can try it with sandboxes or something else, but really nothing's come close to instantaneous hot reload preview like this. Sandboxes cannot do that. So because of that, you should be aware of Web Containers pricing, which is basically free for, I believe, yes, for non-commercial usages. It's completely free. These API sessions, I'm not even sure how much this is, but it never stopped working for me. So I doubt it's going to stop working for your personal projects. But if you plan to commercialize your project, make sure to contact them. So I'm telling this to people who want to build this into a business or something. You should probably check out StackBlitz pricing. I haven't found any better solution than web containers. I think they're an industry standard. If you know something better, feel free to leave a comment. So I will review and maybe teach that in the future. But I mean, you can see how well web containers actually work. So go ahead and play around. ask AI to create something and try and running it in the preview. Perhaps you will see some errors, just so you can see how error screens look and things like that. Other than that, I think we're done. So we completed the web containers, file tree, terminal, and the preview settings. So chapter 14, web containers, terminal, and preview. I'm going to do git checkout-b, chapter 14, web containers. terminal and preview, git add, git commit, 14 web containers, terminal and preview, and git push u origin, 14 web containers, terminal and preview. So we now just pushed a new branch in your IDE. You should now see that you are on your new branch. You shouldn't have any unstaged changes anymore. And we're going to do the usual thing now. We're going to open a new pull request and we're going to review it. So let me go ahead and create a pull request. And now we're going to go file by file to see if we made any critical mistakes here. And here we have the summary by CodeRabbit. We added project preview capability with live development server display. We integrated terminal pane to view server output and commands in real time. We added settings panel to configure custom install and dev commands per project. We added a restart button to reset the previous server. We enhanced security headers for cross-origin resource sharing. And now let's take a look at some issues it found. So the first issue it found is the fact that we are applying these headers across all paths. Initially, I did this simply because that's what fixed my issue. But looking at it now, perhaps we could limit it to specific forward slash projects and then a specific project ID because that is the URL where we actually need that course to happen because of the iframe and the web containers. So in here it's warning us that this might infer with something else we might be doing in the future. So that's actually a very good comment. we might actually, especially for production if you're planning, I would recommend just specifying the specific path where this is needed, where we render the iframe. So that would be this path as CodeRabbit so politely told us. Great. And you can see this is how you would write it. So it gave you the solution, forward slash projects, and then you would do any path after the projects. In here it told me that I'm using tailwind incorrectly that I should be using the important sign at the beginning. You can do that, but it also works this way. I gave it a screenshot showing that it's parsed correctly and after that it stored that information. So now it knows that it works. In this hook, use web container, it noticed that we're not guarding the async boot sequence against restart or unmount races. so yeah we could look into that I'm not exactly sure what it means just at the top of my mind but I will research if this is something critical it does say it is major but I will I will take a look and see you know how exactly we can fix this is it something simple or something we would have to rewrite entirely but you know for this state of the project it works pretty well but just keep in mind that yes there are possible restart or unmount races going on here and in here it's basically making us aware that we are skipping empty files during hot reload and we know that we wrote that on purpose same with storage files simply because they increase complexity and i just wanted to show you that it works at the moment in here it is telling us that there is a potential Okay, so potential issue, we can avoid rendering a stale preview when error is set. So if we have a preview, that's because, yes, we independently render preview URL and we independently render the error. So we should probably not render the preview URL if error exists.\nbecause there is a chance we render both of them at the same time. I think that's the problem, yes, because we independently render this and we independently render this. There's a chance both of them are active at the same time. Good catch. So those are some things to fix. Other than that, pretty good pull request. Let's go ahead and merge the pull request. And once we do that, let's go ahead back to git checkout main. and git pool origin main so we are up to date and as always what i like to do is confirm that inside of my ide so i'm on the main branch and inside of my source graph right here you can see i've detached to 14 to implement web containers terminal and preview and merge that into main i believe that marks the end of this chapter so we have successfully implemented web containers and course, we built complete file tree mounting system, and we implemented the terminal with xderm. And finally, we created preview settings with custom commands. Amazing job, and see you in the next chapter. In this chapter, we're going to implement GitHub import and export functionality. We're going to enable users to import a GitHub repository into Polaris as a new project, and we're going to allow them to export a Polaris project to a new GitHub repository. Both of these features will require GitHub OAuth integration via clerk, a background job processing via ingest, as well as binary file support for which we have already prepared when we started building the schema. So let's start by installing the dependencies needed to make this work. So the dependencies we're going to install are Octokit, which is the official GitHub API client, isBinaryFile, which is a simple NPM package I found, which is very good at detecting whether a file is binary. There are many ways you can do this, many packages. I just found this one to do the job as I wanted to. And React icons simply because I know there's an icon for a GitHub logo there. So let's go ahead and install these three packages. After that, I'm going to go ahead and show you my package JSON so you can see exactly the versions that I have. So is binary file 5.0.7, Octokit 5.5, and looks like we already had the React icons 5.5.0. If you didn't have it, Now you do. Great. So what we ought to do next is actually configure our clerk and add an additional OAuth scope to our GitHub OAuth provider. So using the link on the screen, you can visit clerk's dashboard here. And let's go ahead inside of configure SSO connections. Now in here, you will basically see providers you have decided to add when you configured clerk. If you don't have GitHub here, go ahead, click add a connection for all users and search for GitHub. Once you've added GitHub here, go ahead and open it. I would recommend that you enable it for sign up and sign in. This is what I told you to do in the beginning. So it's very easy for your users to immediately get their GitHub connected. Otherwise, they would have to do it additionally. But it's perfectly fine if you have multiple providers, right? You can have Google, you can have email and password, username, right? But it's very useful to have GitHub as well, since our accounts will be so tightly coupled with them anyway. Now, here's the problem. GitHub OAuth comes with some scopes. But right now, if we tried to use the token from the user who logs in using GitHub, we wouldn't have the necessary permissions to actually load their private repositories. Because of that, we have to enable use custom credentials. And lucky for us, Clark actually shows you the exact documentation on how to add GitHub as a social connection. So we already did this. We navigated to SSO connections. We selected add connection for all users and we added GitHub or we already had it from the beginning. Now to make the setup process easier, they recommend keeping two browser tabs open. One for the clerk dashboard and one for GitHub developer settings. So make sure you have your developer settings open on GitHub. You can use the link from the documentation right here. GitHub developer settings and you should then see all the GitHub apps that you have, all OAuth apps and all personal access tokens. All right so now what we have to do is we basically have to create a new GitHub OAuth for Clark. So let's register a new OAuth here. I'm going to go ahead and select new ALF app. I'm going to call this Polaris. For the homepage URL, for now you can just go ahead and use localhost 3000 because that's where our app is running. But for authorization callback URL, you have to copy exactly what's written here. So let's go ahead and paste that and let's click register application. Now in here you have the client ID which you can immediately copy. You can go back to clerk and you can paste the client ID. And now we need to obtain the client secret by clicking generate a new client secret. This will most likely require you to do two-factor authentication. Once you've successfully authenticated, you will have access to the secret. Go ahead and copy it as this is the only time you will see it. And once you have the secret, you can go ahead and paste the secret here. And now here's the deal with the scopes. Basically right now, as you can see, I have scopes for user email and read user. And I'm going to go ahead and also add repo scope. So at the time of me making this tutorial, this is the scope that is required in order to access users' repositories. So that's the scope we need and you can of course later always add or remove scopes so let's go ahead and click save great now what I would suggest doing is running your app and simply checking if everything still works just fine regarding the login so let's go ahead and see did we implement a way to log out I think we did right here so I'm going to go ahead and log out now I will refresh my page and I'm to go ahead and sign in. I'm going to use GitHub to sign in and let's see. There we go. You can see that now the scopes are a little bit different. So besides my personal data, email address and profile information, it also shows access to repositories, both public and private. So this is what all of your users will see that Polaris is using that scope to access repositories. Perhaps there is a more granular scope, which might be better to use. But for our purposes, a repo is the one we need. Basically, you should see this now. And you can also see that user can also give access to any organizations that they have. Basically, let's just authorize the user and everything seems to work just fine. Great. So that part is officially finished. So what we have to do next is we have to develop some system mutations. Now, most of these system mutations will also refer to something we already implemented. So I'm just going to give you a reminder. Inside of convex schema.ts, if you take a look at projects, you can see that we already have import status and we already have export status, as well as export repo URL. So these three have not been used at all, but we already implemented them. You can see, let me see, in chapter seven projects. That's where we implemented these three fields. So just confirm that you have them. If by any chance you don't, go ahead and pause the screen and add these three fields. Everything else I think is pretty standard. So now we're going to go ahead and we're going to focus on convex's system mutations here. So the last thing we added here are a bunch of agent tools. And now we're going to go ahead and implement all the mutations we need to successfully import a GitHub project or export a GitHub project. So the first one we're going to do is called cleanup. So I'm going to go ahead and prepare it. The cleanup mutation will accept the internal key, such as every single mutation inside of system.ds, and it will also accept project ID. What this will be used for is to very simply clean up all files within a project. Since if you trigger an action to import from GitHub, it might be a good idea to clean up any files you might have created before that. Because an import, in my opinion, is a mirror, right? So it should be a one-by-one replica of what's in the GitHub. So the first thing we're going to do here is we're going to fetch all the files using a project ID, using by project index here. And then what we're going to do is very simply delete the files here, like so. and let's go ahead and simply delete any storage files if they exist and let's go ahead and delete file id so in this scenario i don't think we need to do any recursive deleting usually that's the first thing that i think of whenever we delete file is like hey i need to detect if this is a folder and then delete all of its descendants but not in this case because these are literally all the files, both folders and files. We are not querying by type anywhere, just by project. So in this for loop, all files and folders will be deleted as well as any storage if they were binary files. Great. Now let's go ahead and implement a very simple generate upload URL Generate upload URL is a mutation which will accept the internal key and it will very simply be used to use context to generate upload URL This is from the Convex's documentation on how to upload files, and basically in order to upload a file, you first need to obtain the upload URL from the backend. And in our case, since this will be a background job communicating with Convex, We've put this inside of the system queries, well, system mutation to be specific, and we will very simply return to the ingest background job. Here's the upload URL for any files you want to upload. Those will probably be some images or some fonts, any binary files that the GitHub repository might have. And now let's go ahead and develop the actual create binary file mutation. So, the createBinaryFileMutation will accept internal key, project ID, name, storage ID, and an optional parent ID with an ID of files. As usual, the handler will be validated using validateInternalKey. then we're going to go ahead and we're going to fetch all files by this project's parent id my apologies no that's i i was focusing on the name of the index we are going to fetch all files in a specific project and by a parent id so independently not by project's parent right it's just that the name of this index confuses me so much every time I read it I read it like that and what we have to do here is we basically have to check when we create a binary file this is the exact same thing as if we were creating a new file in a folder so let me show you again if I have foo.js and if I attempt to create another foo.js I get an error I should not be able to do that, right? So imagine if they were binary files. So if this were image.png, for example, and I created another image.png, again, I shouldn't be able to do that. Now, these are not how images are created. Images are going to be created in a binary format, but we still have to check in this specific folder, folder being the parent ID, are there any existing files? So if we can find an existing file with the same name which is a type of file as well let's go ahead and prevent this from happening by throwing file already exists and then we can simply go ahead and create a new file using context database insert files with the project ID, name, type of file, storage ID, arguments, storage ID, and parent ID, arguments, parent ID, and let's initiate the updated ad. And finally, let's return file ID. Great. Now, we're going to go ahead and create a mutation called update import status. So this mutation right here, update import status, which I've just added, will accept the following arguments. Internal key, project ID, status, which needs to match what's in our schema. So let me confirm import status, importing completed and failed. Importing completed and failed. So these need to match. in the handler. We're going to validate internal key and we are very simply going to just patch the project ID. Also, there is a new syntax for this. You can now specifically define which table you want to patch. I think this is better. I like the explicitness. Basically, convexes IDs have a very specific prefix which tells them whether something is a table of project or file so you can't just pass this but they've updated it so that you can explicitly select the table and I prefer this way more simply I think explicitness is always better okay so this is why I just added this function without typing it because it's a super simple one it's just used to update great Now we're having a very similar thing, which is the exact same function as this, but instead of for import status, it's for export status. So it's exactly the same. I'm just going to add it. Let me go ahead and show you. update export status is a mutation which again accepts the internal key project id status which can be exporting completed failed and canceled as well as as well as the repo url which is an optional string so just make sure that all of these here match exactly we are going to validate the internal key and then we're simply going to update again let's do patch projects here arguments.projectid export status arguments.status export repo url arguments.repo url and update it at date.now great now let's go ahead and add a function called get project files with urls so this is going to be a different one so let's just prepare it like so get project files with urls it will accept an internal key project id it will start by validating the internal key so that we know we have access to do this and we're just going to load every single file in the project just as we did in the cleanup function right so using the by project index we're simply loading every single file here. And what we're going to do now is we're going to return await promise.all. And inside of here, we're going to go ahead and call files.map. And then inside for each file, we're going to initiate an asynchronous function, which gives us access to a file. And we are very simply going to check if that file has a storage id we have to fetch that file so let's go ahead and do url await context storage get url using file.storage url because right now if we were to just load all files in a project right if i were to just you know load files here and one of these files was a binary file. I couldn't do much with storage ID. What do I do with storage ID? On the front end, I need some kind of URL, something to display it. So that's what we're doing here. We are kind of preparing these files so that they are readable by the front end. Using the storage ID, which we store in the database. So each file has an option of storage ID, which will allow it to become a binary file. But again, we can't do anything on the front end with the storage ID. But by using context.storage.getURL, we can get an actual URL. And then we're very simply going to go ahead and modify that current file by passing in the storage URL. So the front end can then do something with it. And for all other files, we're just going to return the exact same file and set the storage URL to null. Great. Now let's go ahead and implement a very simple create project mutation. This will be used exclusively for when we import a GitHub project. So we already have, I think, the exact same one in projects, create. It's very similar to this one. I'm just going to add it so you can see the differences. The create project mutation accepts internal key, name of the project, and let me see, owner ID, that's not something we need, I believe, or maybe, okay, we do. Let me see. Yeah, okay. So internal key, name, and owner ID. We're going to validate internal key. And then we're going to simply create a new project with the name, the owner ID, updated at, and here's a catch. The import status will be set to importing. Why? Well, because we know that this specific create project will only be used by Ingeist background job when it's starting to import a project from GitHub. That's why this is inside of a system and not inside of any other one. That's why it's using an internal key to validate because there's no alph here. So yes, that's the only thing. I kind of don't like having such a generic name with such an important status here. But, you know, for tutorial purposes, this is fine right now. You can, of course, change the name later to something more specific. All right. That's all for the system mutations that we have to do. Now let's go ahead and focus on building the API routes. So we're going to start by a import route. Let's go ahead and inside of app API and let's create a new folder called GitHub. Inside import and inside a file route.ts. Oh, I made a mistake. As you can see, route.ts is not inside of the import folder. that means in next js this will not register as a route so i have to drag it inside to make sure it's registered this might trigger some cache invalidation you can just save this file close this file and then close the entire dot next folder so it doesn't distract you let's go back instead of route.ts let's start by importing zod next response from next server out and clerk client from clerk Next.js server. In here I'm going to import convex from convex client and ingest from ingest client. Then I'm going to import API from convex generated API. I'm going to define the request schema for this API endpoint which is just a simple URL And then I going to create just a simple function which will help us parse GitHub URL Now you can decide whether you want to do this or not So function parse GitHub URL accepts a URL string, and it will check if URL matches this specific regex. I don't expect you to write this out. If you want to, of course you can, but you can also just visit the source code and copy it is from this exact location app folder api github import route and again this isn't anything important right it's just something to tell to the back to the user on the front end like hey i think you gave us the wrong url but then again urls might change in the future so i'm not sure how good or bad this is i'm basically going to show you that you can do it both with or without this function so don't worry about this function too much right you can completely choose to skip using this function now we're going to actually export the post request so what i'm going to start is by using await alf which we've imported from clerk and check if we have a user id in case the user id is missing i'm just going to throw a next response unauthorized then i'm going to go ahead and do an await request.json and I'm going to do request schema.parse on the body and here I'm going to have parsed URL. Then what I can do here is I can extract from the URL using the function parse githuburl the owner and the repository. So basically if I go ahead and do github.com, Antonio, my profile name basically, and then the name of the repository like this, this function parse GitHub URL would extract that into an object, owner, this, and repo, this. Right? So you don't really need that, but it is useful, especially since I don't think the URLs are going to change like tomorrow. But again, be careful with this because URLs sure can change in the future. But at this point, there's so many GitHub URLs that they are going to have to maintain them or backlink them or something, right? So this, for example, is a completely valid URL, right? and what we have to do now is we have to obtain our github token so we're going to start by creating the clerk client and then we're going to get all the tokens we have for this user so await client.users get user oauth access token and simply pass in the user id and then for which provider for github i think this is strictly typed yes you can select exactly for which one and then very simply choose the first one. Tokens, data, first in the array, token. In case we are not able to obtain the GitHub token, unfortunately, we can't even begin fetching a URL. So let's go ahead and throw nextresponse.json. GitHub is not connected. Please reconnect your GitHub account. At least that's what we assume the problem is. Now we have to set up our internal key. So that's going to be process.environment. Oops. And let me go ahead and check. I think I've done this a few times before. I have Polaris convex internal key. So I'm just going to copy that, paste it here. And again, if we don't have the internal key for whatever reason, let's throw 500, basically a server configuration error. And now that we have the internal key, we can create a new project from here. So I told you this will happen from the ingest background job, but actually it will happen here in the route. My apologies. So we're going to call await convex.mutation, api.system.createProject, which is exactly what we've created last, I believe. And we have internal key, the name of the repository, which we can just call the project exactly what the repository is called. For example, here, the repository is called cursor dev. So instead of, you know, anonymizing the name or randomizing the name like we do when we create a new project from the projects page, we can just reuse the name here, right? Why not? And we match the owner ID to be the currently logged in user ID. Great. And then what we're going to do is we're going to trigger a background job, which is going to start synchronizing from GitHub to projects. So await ingest.send name. This will be the name of the event which we're going to create. And the data will be the owner, the repository, the project ID, and the GitHub token. And then finally, let's go ahead and just return nextResponse.json, success true, and the project ID. And now while we are developing API routes, I also want to develop the export API route. So let's go ahead and create another route.ts inside. So this one is app folder, API, GitHub export, route.ts. Again, we're going to start with Zod, NextResponse, Auth, and ClerkClient, Ingest from IngestClient, and ID from ConvexGeneratedDataModel. We're then going to develop the request schema for this API. Project ID, RepositoryName. So this is now in reverse. This is for creating a repository on GitHub. So we're going to have some limits here. If you want to create a new repository, the name needs to be a minimum of one character and a maximum of 100. The visibility can either be public or private, and we're going to default it to private. The description can be maximum of 350 characters, and it's optional. Now, let's go ahead in here. And let's start by obtaining the user ID. let's check if we user id is missing and throw 401 then let's go ahead and get the body let's go ahead and parse project id repository name visibility and description from the body so we're using request schema.parse body then let's go ahead and initiate the clerk client let's go ahead and get all the tokens we have for GitHub. So the same thing we just did in the previous route. And let's get the GitHub token from the first one in the array. In case the GitHub token is not connected, let's go ahead and throw an error. Let's prepare by copying the internal key from here. Adding it here. Then doing a check if the internal key is missing. And throw 500. it. So since we just moved between these two, just make sure you are developing the export one. Don't accidentally override your import one. Great. And what we have to do now is just send another background job event. So this event will be await ingest.send, github export.repo with data, project ID, project ID as ID projects, repository name, visibility, description, GitHub token and the internal key we can see, we can decide if we want to pass the internal key as a prop since we can just easily check it there but in here it's kind of already checked so we don't have to do double checks but we'll see and I don't think we need to do any typecasting here actually yeah okay and then we can just go ahead and return something like next response.json success true, event id event ID is zero. So this event ID is not important. For example here I don't think we even threw it. So you can decide. If you want to be specific so that your network logs show the event ID you can copy so the same appears in the import one. And let me just go ahead and check if there are any more things that are not missing so success through project id i think in here we're not passing project id great so i just want to make them the same since they are so similar already okay so now let's go ahead and implement a cancel route so far we've implemented the export and import now let's give the user ability to cancel. So another route.ts. And at this point, let's just copy one of the other ones. So I'm going to copy the export one. And I'm going to paste it here. Actually, I'm going to import since it's more similar. So copy the import route, go inside of cancel, and paste it here. so we need zod, next response we're not going to need clerk client you can remove that, suggest out convex, ingest API and we're also going to need id from generated data model we're not going to need the function parse github url the request schema will not accept the url but a project id the out check will be exactly the same. The parsing will parse for project ID. We can skip the entire check here and go immediately down to the internal key. And let me just see what's the problem. Cannot redeclare blocked scope variable project ID. that's odd because it appears later, no worries so we were here, we check if we don't have the project the internal key and we're not going to be calling any convex mutations I mean here but we are going to send an event so let just go ahead and send an event called GitHub export dot cancel And the data is just going to be project ID. So basically, we are going to give the user ability to cancel an export. right so if the user attempts to export it i'm not sure how long that's going to take right you can it can have thousands of files right so for them not to be in a forever stuck state we're going to give them the option to cancel now we could develop the same for import but the user can just delete the project because we want to give the user option to cancel when they are exporting since that is a project they have developed in Polaris. So we need to give them a way to get out of this situation. Whereas with import, it's a brand new project. Nothing of value will be lost if they just delete the project. But of course, you can just develop the same once you see how we're going to do it. All right. So once we send the cancel event, we also have to update the status to cancel. So this is where we are going to do the mutation. So await convex mutation. API system update export status to cancel with the internal key and with the project. And let's go ahead and just, Let's just return success true. If you want to, you can do the event again. Simply so we are consistent in all three. There. Okay, so that's the cancel route. And then let's go ahead and copy the cancel route and do the last route here, which will be reset. in reset route.ds we're going to need zod next response alph convex not ingest api and id the request schema is the same alph check is the same parsing is the same internal key check is the same we're not going to be triggering any events and we are very simply going to clear export status. So this will again call update export status and we're just going to reset everything. So status will be undefined and repo URL will be undefined. So what exactly is this used for? Well, the difference between cancel and reset is that cancel is simply used to allow the user to stop the export. The reset is used if the user wants to once again export their Polaris project, but maybe to another repository, right? So there is no synchronization between GitHub and Polaris. There is only manual synchronization. It's not going to be an ongoing thing, right? They are two different separate entities. So that's why we allow users to also just reset entirely if they want to export to GitHub in five different repositories, like, sure, do it. All right. Now let's go ahead and start building those ingest functions. So I'm going to go ahead inside of source, inside of features, projects, and in here, I'm going to create a new folder, ingest. and inside of that I will create import github repository.ts and while we're here I think it might be a good idea to just run npx convex dev simply to synchronize all of those functions and simply to catch if there is an error in any of them in my case there was no error so convex functions are ready if you're seeing an error it might be a good idea to go back and fix them So all of the functions which we've written today is inside of system.ts. So it has to be somewhere here. You can see my green line here, meaning all the newly added ones are these. The cleanup, the create binary file, the update import status, update export status, get project files with URLs, and create project. So if you have any errors, they're going to be here. All right. Now let's focus on the ingest function. We're going to import KY. We're going to import our newly installed Octokit. Same with the binary file. And non-retriable error from ingest. We're going to import the convex client, the ingest client, and API and ID from convex generated. Let's create an interface for this event. So what do we expect to receive? So when we are importing, we need the owner, We need the repository. We need the project ID to which project we are going to synchronize this entire repository. And finally, we need the logged in users GitHub token. So once we have those things, we can go ahead and create an ingest function. Let's go ahead and start with some configuration. The ID will be import GitHub repo. And let's first do on failure. So what should happen when this fails? We have an event and step. And the first thing we're going to do is just check if we have the internal key. So I keep forgetting the name of my internal key here. Polaris convex internal key. So const internal key process.event.environment Polaris convex internal key. if there is no internal key let's just return now what we're going to do is we're going to import the project id for which importing just failed to so you can extract project id from event dot data dot event dot data usually it's just event dot data but when working within on failure this is how you get the project ID. And in here I'm casting it as this event simply so we have the ID of convex here. And then what we're going to do is run a step called setFailedStatus. So at this point, importing has failed. So what I'm going to do is I'm going to await convex.mutation, API.system, updateImportStatus, pass in the internal key, the relevant project ID, and set the status to fail. So whatever goes wrong in this background job, after several retries of course, will simply trigger this on failure, which is going to call the system mutation to indicate to the user, hey, we failed importing this repository. So they can try again. Great. So now that we have that, let's go ahead and define an event name, which is github forward slash import dot repo. What I would suggest you do is you do a search throughout your repository here. You can use command shift F or control shift F while it's highlighted. So it opens like this and just double check that inside of your source app API GitHub import, you have the exact same name for the event that you trigger. So github forward slash import dot repo, it should be a one by one match here. Great. And now we build the actual background job. Of course, it's going to be asynchronous. And let's start by simply extracting the data we need. So owner, repo, project ID, github token, and event dot data as import github repo event. then I'm just going to go ahead and copy the internal key situation here. I'm going to paste it. I'm just going to slightly modify it by throwing a non-retriable error which will simply say Polaris convex internal key is not configured because if that's not configured, the background jobs, sorry, it's throw new non-retriable error. The background jobs cannot do any convex mutations. So now let's go ahead and establish Octokit by passing the alph property as GitHub token, which we extract from a clerk, which we now have with the proper scope because we configured it at the start of this chapter. Great. So now we're going to use the first mutation we've created today. We're going to clean up any existing files in the project whenever this import background job happens. So let's go ahead and run a step called cleanup project, which will call await convex.mutation API system cleanup, and then it's just going to pass the internal key and the project ID because that's all we need. Once it's been cleaned up, let's go ahead and fetch the entire repository tree. so a step called fetch repo tree will have a try and catch method inside in the try method we're simply going to go ahead and extract the data from await octokit.rest.git.get tree using the owner, the repository, 3sha, main, and recursive 1. The 1 is not an integer, it's a string, like so. And let me just see the problem here. oh it's my apologies tree sha this can be main or it can be master depending if it's an older one well i mean technically you could also add an input so the user enters exactly which branch they want to clone but what i'm doing right now for tutorial purposes is just fetching the main So I'm just gonna either try with main or I'm gonna fall back to master.\nLet's return data. And then we're simply going to do the exact same thing in the catch by trying with master branch. Because some older projects might use master. So we're just trying to fetch the tree now. All right. That is that step done. What we have to do now is we have to sort the folders by depth. So parents are created before their children because we first need to create folders before we can start creating the rest of the files. So, for example, the input we're going to receive, the input being this, right? The data here is basically this kind of structure, path source forward slash components, path source, path source components UI. and the output that we need to create is path source first, path source components next, and source components UI last. So we can't basically allow source components to be created because we don't have their parent yet, right? This is all coming back to our schema structure and our parent ID reference. So let's go ahead and start developing this. folders will be tree.tree.filter get the item check if item.type is tree and if we have item.path then let's sort check if a depth has a.path if it has let's do a.path dot split forward slash dot length or zero. Then let's go ahead and duplicate this, change this to B depth and change the variable to use the B variable. B dot path, B dot path dot split. And let's return A depth minus B depth, which will basically sort them as we've defined above. Great. Now that we have the folders, let's go ahead and do the following. We're going to return the folder map from the step so it can be used in subsequent steps. This is ingest specific. So ingest serializes step results. So we must use a plain object instead of map. so what I'm going to do now is try to define a folder id map within a step so await step.run create folders async and now I will create the map but just by using a normal object so map will be a type of object which accepts a key and on the other end it's an id of the file now we're going to go ahead and first go through the folders so for const folder of folders if folder path is missing let's just continue forward there's nothing we can do here but otherwise let's go ahead and prepare a few things path parts which are folder.path.split So basically, when we have things like this, it will be split into source and components. The name will basically be the first one in the array, so we pop. And the parent path, which will be the rest. So we are separating the name, basically. And the last one we need is to find the parent ID. So the parent ID will check if we have parent path, and it will look through the map and check if it's there or mark it as undefined. And then let's go ahead and get the folder ID by creating it. Await convex.mutation api.system.createFolder. Passing the internal key, project ID, name, and parent ID. and then we're going to add to our map for that folder path the equivalent folder ID we have created in our database. So then in the next iteration, if that repeats, we will find that because of this. We will now find that parent path in the map. So we know, okay, the parent has already been created. We can now start importing the children, right? So that's kind of the tricky part here is because Octokit just returns us a bunch of files and structures, but we have to create them in a way that we create parents first and then their children. So that's why we're doing this somewhat complicated logic. All right. And of course, return map. then let's go ahead and solve the binary files so these are a bit more complicated so let's start by getting all files or blobs from a tree all files tree.tree.filter check if item type is blob and we have item.path and item.sha. Then let's go ahead and create the files. So a step called create files. We're going to go ahead and go through each file in our all files filter here. In case file path or file shy is missing, we're just going to continue. And then we're going to open a try and catch block here. Let me go ahead and fix the indentation. Okay. Inside of try, we're going to go ahead and get the blob using the file SHA and owner and repo. I'm not sure how to pronounce it. So when you hear me say SHA, this is what I mean. All right. So how do we get the blob? By calling octokit rest git get blob function, which needs the owner repository and the exact file sha. And that will basically give us back the blob. Once we have the blob, we can go ahead and create a buffer using buffer.from and using the blob content with base64. And then, once we have the buffer, we can call our isBinary function, isBinaryFile, and pass in the buffer, isBinary from isBinaryFile. This will allow us to decide how we store this file. now we also have to define if this is inside of some folder so we have to check for path parts using file.path.split let's get the name of the file using pop let's go ahead and get the parent path using parts.join and then let's check if we have the parent id so very similar to what we did before. If we have parent path, check if we have stored it inside of a folder ID map. Otherwise, mark it as undefined. So this folder ID map, I believe, is returned from here, right? So now, let's check. If a file is binary, we have to upload it. So let's first get the upload URL. We can do that using our system mutation. So upload URL is obtained with await convex mutation, API system, generate upload URL, and it just needs the internal key. Then we can go ahead and fetch that URL. So we will get back storage ID, which we can store into the database by making a post request. So let's go ahead and use ky for that, .post, to upload URL, which we've just obtained. So this is a signed upload URL from Convex, so we can safely upload here. It needs to accept specific headers. So these are the ones it needs. I'm going to show you. Content type, application, octet stream. And the body will be the buffer. So we are now uploading a binary file. And we are returning back in JSON format, storage ID, which will be a type of ID underscore storage. And once we have the storage ID, we can go ahead and do await convex mutation API system, create binary file. pass in the internal key the project id name storage id and parent id so this function was already created in the beginning we are now using it through a background job to actually create that binary file after it's been uploaded to convox through a secure background job using a secure signed upload URL. Great. So that's the case for if is binary. But if it is not binary, in that case, we're simply going to convert the buffer to string using UTF-8. And then we're just going to call a normal API system create file with internal key, project ID, name, content, and parent ID. And in the cache here, we're going to go ahead and do console error, fail to import file, file.path. Great. And then let's go ahead and do one more step, an easy one. Set completed status. It will run await convex.mutation, API system update import status. at this point we have already finished importing, we have finished uploading binary files, we finished creating normal files, we created parents first and assigned everything properly. So at this point we can just call our system update import status mutation with the status of completed Great And let go ahead and return it a very simple success true and project ID Great So that is our import GitHub repo I will admit it was a bit complicated, even though I was getting lost a bit here. So if we made some mistake, we will test that out once we implement the UI so we can actually fire this. But I think it's mostly okay. I think mostly we did everything right. All right. So the next one we have to do is create a very similar background job, but for exporting a GitHub repository. The export ingest background job will be similar, but not too similar. So I'm just going to start a completely blank file. So inside of ingest folder here, I'm going to create export to GitHub, export to GitHub.ds. and the imports are quite similar. So those are KY, Octokit, Non-Retriable, Error, Convex, Ingest, API, and ID. Now let's go ahead and add the interface Export to GitHub event. Export to GitHub event will have a project ID, repo name, visibility, public or private, optional description, GitHub token, and I don't think it makes sense to pass the internal key. So let's remove that. Then let's create a specific type called file with URL. It has an underscore ID of ID files, underscore creation time, which is a number, project ID, optional parent ID, name, type, file or folder, content, storage ID, and storage URL. Now that I look at it, I think we can do this in a better way. I think we can do, let me just see. This will be a type of, I mean, I'm not sure. Let's see. Type file with URL to a type of doc from data model. So let me show you that import right here. doc from generated data model. And we import, we use files. So right now I think they would have like almost a 99% match. The only thing we want to extend with is the storage URL. I think these are now exactly the same. I think. I think this is a much more elegant way of defining that. Okay. And now let's go ahead and create the ingest function export to GitHub. Let's start with the ID export to GitHub. Let's define the cancel on event. The cancel on events will listen to github forward slash export dot cancel and okay so we cannot use match let me see what is the name of the new one i mean we can use match but it's deprecated i want to teach you that uh so let's see we have to use if and i think i have to check data project id matches data project id but not exactly like this so let me check if all right so we have one example in process message dot ds so let's use it to learn we have event dot data and we have async dot data so that's what we have to check for if event dot data is equal to a project id and async dot data is equal to a project ID and they are using double T's. Okay, I think we've done this correctly. All right, besides cancel on, we're also going to have on failure. Now on failure is very similar to the on failure in the import one. So let's go ahead and revisit it. I'm going to scroll up here to find the on failure. Where is it? Here it is. We're going to start by checking if we have the internal key. And if we don't have it, we just do an early return. and now let's go ahead and let's destructure the project id from event.data.event.data and let's cast it as export to github event and then what I'm going to do is call a function very similar to this one so an entire step called setFailedStatus setFailedStatus call convex.mutation instead of updateImportStatus is going to be updateExportStatus and I believe the queries are exactly the same the internal key, the project ID and the status so yeah, I think this works just fine now let's go ahead and let's define the event name so the event name will be github export dot repo again i recommend searching through your project and confirming that this is the event that you trigger inside of app folder api github export route dot ds so github export dot repo So it should match exactly. And it should also match this one. So check this, GitHubExport.cancel. Search for that too. Inside of your source app API GitHubCancel, you should have that event here as well. Great. So you don't have any misspellings. And now let's go ahead and actually build the function. It's going to be an asynchronous function, which accepts event and step. Let's go ahead and start by extracting everything we need. So project id, repo name, visibility, description, and github token. We don't need the internal key. Then I'm going to copy from import a check of the internal key once again and a throw of a non-retriable error and I'm just going to paste it here. So we're going to attempt to get the internal key. If it's missing, simply throw a non-retriable error. Great. Let's start by running a step which will change this project's status to exporting. So set exporting status will call convex mutation API system update export status, internal key, project ID, status, exporting. then just as we did in the import background job we're going to initiate the octokit using auth and github token which we obtain let's go ahead and get the authenticated user using the octokit so we can run a step get github user and return await octokit rest users get authenticated. This will basically return whatever user has passed the GitHub token to. And let's go ahead and alias it as user. And now let's start by creating a new repository without an init so we have an initial commit. So this step will be called create repo. And in here, what we're going to do is return await octokit rest repos, create for authenticated user. Pass the name, repo name, description to be description or a very simple exported from Polaris. Private will be if visibility is set to private and auto init will be set to true. Great. then let's go ahead and wait for GitHub to initialize the repository simply because auto initialize is async on GitHub site so I'm going to sleep for three seconds if this step fails too much for you you can increase this to five or six seconds but most of the time this works with three seconds in fact I mean this is what my debugging led me to this is my conclusion that that's what happens because I had some annoying bug with this. It could be that I was doing something wrong at the time, which I fixed later because this never happened again. But still, I want to show you that this is an asynchronous function. So the next step is basically to get the initial commit SHA, however you pronounce it, right? But we can do that if the repo has not yet initialized, right? So that's why we want to avoid any errors. But still, a background job will retry itself if it fails. So this isn't too important, right? All right. So yes, next step is to get the initial commit. We need this as the parent for our commit. So let's go ahead and create a background job called get initial commit. Make sure it's an asynchronous function. and basically what we're going to do in here is again call octokit sdk so await octokit rest git get ref owner user dot login remember user comes from a previous step get github user in which we return get authenticated and we await it and we alias that to user so now we can use it here, repo name, ref heads forward slash main and return ref object SHA. Now let's go ahead and fetch all project files with storage URLs. So this will be some binary file things, right? Let me go ahead and prepare this step. Fetch project files. And in here, we are going to do the following. We're going to return, open parenthesis, await, convex.query, API system get project files with URLs passing the internal key and project ID and cast this as file with URL and then an array of those So it looks like since we not getting any errors here looks like this cast is correct. I think if I change this to something, you will see that this then causes an error, which means that we have correctly extended the storage URL part. So yes, If you remember, this system function basically loads all the files in a project, and it simply makes use of that storage ID by turning it into a URL. So in this scenario, when we want to export those binary files to GitHub, we need to convert them to a URL that GitHub can, well, turn into a binary file and upload onto their system, because GitHub can't do much with our internal storage ID. so that's why we're doing that. Now that we have all the files ready we have to do the reverse of what we were doing in the import background job. We have to build a map of file IDs to their full paths but luckily this is actually a little bit simpler than doing the other thing. All right so build file paths function accepts an array of files with their storage urls if there are any we're gonna prepare a file map using new map it will have an id of files and object file with url and then let's go ahead and run a quick files dot for each file and simply set it in the file map mapping their id with their content inside. Then let's develop a method inside to get the full file, the full path of a file. So this method will accept an entire object of file with URL and it will return a string. First things first, if there is no parent id, we return a file.name. So the point of this function is to return things like source, components, index.js. But in case it's a root file, it's just index.js. That's why we just returned the file name if there's no parent. Otherwise, open backdix in this return statement here and simply call getFullPath again with the parent. And then forward slash file dot name. Now let's go ahead and just see the problem here. My apologies. Instead of get full path here. So after we check if there is no file parent ID, we have to get the parent of course. So file map dot get file dot parent ID. then if there's no parent return file dot name my apologies i'm kind of getting lost these files are way too similar all right so in order to get full path we check if there's no parent id and do an early return otherwise if the we attempt to get the parent from the file map using file parent id because we map all the files with their ID and their content. And if it doesn't exist there, we again return file.name. So this is kind of an edge case, right? Otherwise, we recursively call this function until it generates the full path string. So it will either early return or it will continue generating the string. Great. Now that we have that, let's go ahead and prepare an object which will store all the files with their paths. So files for each file simply assign to the paths object their full path and their content inside and return paths. Then outside of this function, let's go ahead and actually get all file paths by calling build file paths. Then let's filter to only actual files, not folders. So file entries, object.entries, file paths.filter, skip the first argument, which is the ID, and go into the object and access file.type comparison for file. If it's true, it's going to filter out all of those which are not files. And then if that length is zero, let's throw a non-retriable error, no files to export. And now what we have to do is we have to create blobs for each file. Let me fix the typo here. So let's go ahead and prepare this function. create blobs so the blob has the following structure path string mode type and sha if you're curious about this magic number right here feel free to google that along with blob so you will see a more in-depth explanation but basically this is the mode that makes it exportable to GitHub file system. So simply create an empty array and give it a type of this. It's important that you don't forget this array type at the end. All right, so now that we have file entries and file paths, let's go ahead and do a for loop. So for each path and file of file entries. Let's prepare their content to just be a string. And let's go ahead and prepare their encoding. Are they content? Are they textual content? We're going to use UTF-8. Or are they a binary file? In which case, we're going to use base64. So first things first, if file.content is not undefined, that means this is a text file. So let's do content equals file.content. Else if we have file storage URL, this is a binary file. Fetch and base64 encode. so let's go ahead and do that now we can get a very quick response using ky.get on file storage url once we get the response we can turn that into a buffer using buffer from await response.array buffer and then we can store that into content by turning it into base 64 and let's go ahead and set the encoding to base64 in that case. Else, so if it's not a text file and not a binary file, skip files with no content at all and continue. All right. Now let's go ahead and actually create the blob using Octokit. So await Octokit rest git create blob owner user login repo content and coding data blob and then let's go ahead and push to our items array path again the same mode as before type of blob and sh a blob sh a and let's return items there we go then let's check if tree items dot length is zero it means we failed to create any file blobs we didn't export anything so we throw back because this is a non-retriable error right something went very wrong up there but otherwise we are ready to start creating the tree so this is very simple we are going to create a step called create tree and simply return octokit rest git create tree owner user login repo repo name tree tree items then we have to create the commit with the initial commit as the parent so we are now creating a new commit right we've just pushed these files and we have to commit that. That's how Git works. So again, just a very simple Octokit SDK function. Octokit.rest.git create commit. Owner, repo name, message. It can be whatever you want. We're going to use initial commit from Polaris. Three and parents, which is initial commit SHA. Great. now let's go ahead and let's update the main branch reference to point to our new commit that's again going to be a simple octokit function so a step called update branch ref return await octokit rest git update ref with the owner repository ref heads main shacommit.sha and force to true. And then last step here, set status to completed with repo URL. So set completed status, convex mutation, API system update export status, internal key, project ID, status completed, And finally, the repo URL using repo.html underscore URL. So we know we have access to this from the start, actually. It's just empty, I believe. So let me find where do we get the repo from. Here it is. Great. All right. That is it. All we ought to do now is just a simple return here. There we go. Success true. Repo URL. and how many files were actually exported. Great. So that is our export to GitHub function Now we have to register these ingest functions For that we have to go inside of app API ingest route And let's go ahead and pass in import GitHub repo, export to GitHub. And that's it. And at this point, we can remove demo generate and demo error. Great. Now let's go ahead and add some UI so we can actually test this. So we're going to go inside of source, inside of features, inside of projects, components and I'm going to go ahead and create a new file called import github dialog.tsx. I'm going to go ahead and import KY and HTTP error from KY. I'm going to import ZOD, Sonar, Toast, Use Router from Next Navigation, Use Form from Tanstack React Form, Use Clerk from Clerk Next.js. I'm going to import Button from Components UI Button, Dialog, Dialog Content, Description, footer, header, and title from components UI dialog. And then I'm going to import input from components UI input, field, field error, and field label. Then I'm going to import ID from convex generated data model. Let's go ahead and define the form schema. It's very simply going to ask the user for the URL they want to import. As simple as that. And thanks to our function, let me go ahead. Is it here in GitHub import? I think it is. It is parse GitHub URL. So this is why I've kind of developed it. So we allow the user to just enter a URL, right? And then on the back end, we're just going to extract the owner and the repository, right? So I think it's a useful function. feel free to copy it from the source code simply because this is a bit heavy to copy. So this will make it easier on the user experience so we do the hard work for them. All right. So the props for this component are going to be open and on open change. And let's go ahead and export the component. So import GitHub dialog accepts open and on open change. Let's prepare a router from useRouter. and let's prepare open user profile from use clerk. So we're going to use this in case we get an error that user doesn't have their GitHub connected. Because remember, while we do allow GitHub login, you can also enable Google login or a bunch of other ones, even email and password, right? But the good news is user can always connect additional ones from their account settings. clerk makes that very very easy now let's go ahead and define the form the form will start with default values which is just empty url the validators are going to be on submit which will simply look for form schema and then we'll develop the actual on submit asynchronous method which gives us access to the value and what we're going to do is we're going to initiate a post request to so let's do the following. Let's extract project ID from await ky.post forward slash API GitHub import. Don't misspell this. This is not type safe. You can write whatever you want here. Just make sure that you actually have API GitHub import. So no typos anywhere. all right and as the body of this post request we're going to add json url value url so exactly what user writes we're going to parse this back as json and let's go ahead and write what we expect which is success boolean project id id of projects and let's see what else do we expect also the event ID which is a type of string. Alright. So let's be correct for our frontend here and write exactly what we expect back. So after project ID it's event ID which is a type of string. Great. At this point we can already send toast.success importing repository since it's a background job. So it didn't finish it just started. And let's go ahead and close this dialog. And let's reset the form. All right. And after that, what I want to do is also do a router.push to projects project ID. Simply so we can immediately, well, redirect the user there. Let's wrap this inside of a try and catch. and in the catch method, let's go ahead and grab the error and let's check if error is instance of HTTP error. Let's go ahead and extract the body of the error using await error response JSON. Let's go ahead and add types here, error string. And now what we can do is we can check if body?error?includes GitHub not connected. In that case, we can throw toast error GitHub account not connected. and an action with a label of connect and an on click open user profile. So if there is an error happening here, user will simply get a subtle toast which says GitHub account is not connected and an action. So down here, that's where it's going to happen. and an action to open clerk user profile, which will basically just trigger manage account here, which will allow the user to then connect another account. All right. And at this point, so what I'm going to do is just indent my thing in the try here. And for this part, GitHub not connected. So again, be careful here. Do we throw that error? Here it is. GitHub not connected. For example, here's the bug. I'm not sure this would work now because my GitHub here is capitalized, but it's not here. So make sure to capitalize it here in the includes because it's checking for a string. You can see it's a fragile way to do it. You could do it with a very specific status or maybe adding code here which could be an enum like github missing and then you can check for that instead of a string but of course that is for actual you know production problems for now just for tutorial's sake we can just check for a string but just to make it work make sure you're actually throwing that part of the string so you can actually catch it here all right so after we throw that error, we also ought to close the dialog as well. Great. And then finally, outside of this if clause here, let's just do toast.error, enable to import repository. Please check the URL and try again. So there's a chance that the error is unrelated to the GitHub account. Since we do an early return here, we don't have to put this instead of an else, right? All right. And now let's go ahead and actually build the UI. So let's go ahead and add a return here, dialog open and on open change. Let's go ahead and add dialog header, dialog content and dialog title. Then let's go ahead and add dialog description. Enter a GitHub repository URL to import. A new project will be created with the repository contents. For the actual form, we're going to go ahead outside of the dialog header and define the form element with on submit, prevent default, calling form handle submit. Inside, let's go ahead and do form.formField. So what's the deal here? I don't think I've explained this previously. So form in this case is just a native HTML form element. Form in this case is referring to this form. So don't be confused about that. So this form.field is not native HTML. This is a specific component that's being exported through the hook. That's why we can access it this way. it just coincidentally perfectly matches with this i actually like it but it is a bit confusing when you don't understand what's going on because this also feels like it's native it's not this is a hook all right so form.field with the name of url will have access to that field properties in this way and then in here we can immediately check if it is invalid so we're going to store the invalid state using field state meta is touched and if not field state meta is valid. Alright. Now we can finally return how the field is going to look like. So let's do field data invalid. So basically just an accessibility attribute here is invalid. And let's go ahead and define the field label. Again, HTML4, so it has an accessibility attribute. The name is, I mean the actual label is repository URL. And then let's go ahead and define the input. The input has ID of field name, name of field.name, value of field state value, on blur, field,\nhandle blur, on change, field, handle change, event, target value, aria invalid is invalid, and the placeholder explaining to the user the structure that we expect. So then our backend parse GitHub URL function will extract the owner and the repository and pass it as separate objects or keys, should I say, to relevant ingest background jobs, which call the OctoKit further on. In case there's an error in the actual field, let's display that by checking if is invalid and rendering the field error and passing the errors with field state meta.errors. Great. So that marks the end of the form field. All we have to do now is create the dialog footer with a margin top 4 class name, which will simply give us two buttons. The first button will be a type of button. This is very important. So this button is not used for submitting. That's why we explicitly give it a type of button, a variant of outline and on click, on open change, false. Basically, this is used to close the model, like cancel it. And for the submit one, we're going to access that through another form.subscribe element. The subscribe element will have a selector, uses state, and returns an array. State can submit, and state is submitting. Then we can work with those two fields using the following syntax. So again, can submit and is submitting. and we are very simply going to return button type submit, which is disabled, if not, can submit, or is submitting. If it is submitting, display importing, otherwise import. So to make it easier for you to read, I'm going to collapse it a bit. There we go. Great, that is it for import GitHub dialog. and now the last ui component we need to create before we test this out is the export popover so i'm going to go ahead and copy and paste this since they are somewhat similar and rename the copy to export popover.tsx double click to make sure you are working inside of export popover and let's go ahead and start by checking our imports. So I'm just going to add an overall import for React since we're going to need it. So React from React. KY is good. Zod, Toast. We're not going to need useRouter so we can remove that. We will need useForm and useClerk and for the icons, we're going to need check, check icon, check circle to icon, external link icon, loader icon, and x circle icon. For the components, we're going to use button. We are not going to use dialog. Instead, we're going to use popover. So popover, popover content, and trigger. We are going to be using the input that can stay. And we're going to have field, field error, and field label. And we're going to have two more components besides that. We're going to have select, select content, select item, select trigger, select value, and text area. Then I'm also going to import a hook called use project from hooks use projects. We already have ID from generated data model. and let's also add an icon from react-icons forward slash fa fa github. All right now let's modify the form schema. So the form schema will have a field called a repo name. Repo name will be a string with a minimum length of one maximum of 100 and a regex for only alphanumeric characters hypens underscores and dots basically the same rules that github enforces so a very simple regex here you don't need to add it but it will prevent the user from trying to submit an incorrect one for the visibility prop it's going to be an enum of public and private and then the description which has a maximum length of 350 which is too long after that all right for the props the only thing we're going to need is the project ID and it's going to be called export popover props. Then let's go ahead and change the export instead of import GitHub dialog to export popover. Export popover simply uses the project ID and the same named props. Since we don't have the router hook, we no longer need it. But instead of that, we can add the project and load it. And we can define a simple use state from react.useState, open and set open. And we can leave the profile here. Then let's keep track of the export status of the project and export repo URL of the project. So this way we can track since Convex is a real-time database, what's the current status of the background job and did we receive a final repo URL we can visit. Now we go to the form. So the form will have three different values here. Repository name in which we are going to attempt to load the current project's name. But since there are different rules for what we allow users to name our project and what github accepts we have to use dot replace and only accept alphanumeric characters dashes hyphens and dots if you want to you don't have to do again this regex at all you can just do a fallback like this but this will prevent any problems from happening visibility will fall back to private and we're going to cast it as the only two enemies we accept and description will be empty validator's object stays the same and now in the on submit it's going to be a little bit differently so in here we're going to call API GitHub export, again make sure you didn't misspell this so just double check inside of your import, my apologies, inside of your API GitHub export export. So the JSON it accepts is a little bit different. It's not URL. Instead it's the project ID repository name which is value.repo name. Visibility value.visibility description value.description or undefined and this will not be needed at all. There we go. So now there should be no problems here. For the JSON, we don't really care. We don't have to. We're just initiating. We don't really care about the result itself. And we can remove... well, I guess we can just leave the toast message, which would say, export started. I think we can kind of send a success message. And then immediately let's go into catch and make sure to check for the exact same error, GitHub not connected, and allow the user to connect and change this set open to be false. The reason we are not doing set open false here is simply it's a different UI. You will see. But just in case you were wondering like, hey, why are not we closing it here? Because we are closing the import one because that's a dialog. This is a popover. So it's a little bit different. Okay, what I want to do now is just double check that this error actually works. So for that, we're going to go instead of export route.ts and check GitHub not connected. Make sure you're throwing this. Make sure the capitalization is correct. Make sure it's the exact same line you're checking here. Great. So instead of those errors saying unable to import repository, it will be unable to export repository. Unfortunately, this is not due to the URL. It can be many things. So we're just going to say unable to export repository. All right. For the return here, I am... Okay, I'm not going to delete anything just yet because there are a few more functions we have to develop. The first function is handle cancel export, which is basically a button to cancel the export. So it's going to call API GitHub export cancel. Make sure it actually exists. API GitHub cancel. And now that I look at it, mine is actually incorrect. So my cancel route is in a different place here. So yes, I'm going to drag my cancel route and put it inside of export. Because that's where I meant to add it. So yes, it should be API GitHub export cancel. My apologies, I think I've missed this completely. All right, so now this makes sense. API GitHub export cancel, allowing us to cancel an export. Then let's go ahead and add a function to reset the export. Again, I think we're going to have to move this. So API GitHub export reset. Let me see inside of my API here. Yes, let's move reset and put it inside of export folder because that's where I meant to do it. I just completely forgot my apologies. So yes, because both of these entirely refer to exporting. Perfect. Now let's go ahead and develop a function called render content. If export status is exporting, In that case, we're just going to go ahead and display a div with class name flex, flex call item center and gap three A loader icon with class name size six animate spin and text muted foreground a paragraph with text exporting to GitHub text small text muted foreground as the class names and finally a button to cancel it. So this button will have a size of small, variant of outline, class name of width full, and on click, handle cancel export, and the label cancel. All right. So that is for that case. Now let's do if export status is completed and if we have export repository URL. In that case, let's go ahead and copy the outer div since that stays the same. The only thing we're going to check is the icon which will be check circle to icon size 6 and text emerald 500 to give it a nice greenish color beneath a small description repository created with text small and font medium class name beneath that another text text extra small text muted foreground text center your project has been exported to github then let's go ahead and create a div class name flex flex column with full and gap two. In here, let's go ahead and add a button to open that GitHub repository. So this button right here, size small class name with full as child property. Inside an href with a target forward slash blank, my apologies underscore blank. I think we can do this with a normal link though. Let me see. We just have to import link from next link. I think this should work just fine. Yes. And external link icon icon and view on GitHub label. Now we're also going to add a button to reset the entire thing, right? So once the link is shown, view on GitHub, next to it, This button will serve as the reset button. And by reset, we don't mean we're going to delete it from GitHub. No, the user now knows, hey, that's the link. Go on your GitHub and maintain it there. But click this button if you want to change the repository, right? If you want to export it again to some other place. So that's button size small variant outline class name with full on click handle reset export with a close label. Great. Now, in case the export fails, we need to display an error in that case. So we're going to display something very similar to the first one, to exporting. So let's just go ahead and copy this entire thing here. And let's just paste it here. Instead of loader icon, it will have X circle icon. It won't have animate spin. Instead, it will have text rows 500. Then for the paragraph, we're just going to say unable to export with text small and font medium. And beneath that, text extra small, text muted foreground and text center. Something went wrong. Please try again. For the button, it will have a size of small variant of outline with full, and this will be handle reset export. So if the export fails, we're going to allow the user to trigger a reset from here as well so they can enter new information rather than just try the same thing again. all right and then finally in the return we're going to go ahead and build our form so let's go ahead and build form here with an on submit prevent default and form handle submit let's go ahead and add a space y4 inside space y1 let's add a heading export to github with font medium and text small beneath the heading we have a paragraph text small and text muted foreground export your project to a github repository outside of that div we're going to add our first form field which will be used to enter the repository name now to access the field property we use the following syntax and then in here what we can do is we can extract the is invalid into a constant by checking for field state meta is touched and field state meta is valid. Great. Then let's go ahead and actually return the field. So we're going to use the field component and give it an accessibility attribute data is invalid. We're going to add it a label which says repository name and the accessibility HTML4. And let's go ahead and render the actual input with the ID of field name, name of field name, value of field state value, on blur, field handle blur, on change, field handle change with event target value, another accessibility attribute for is invalid, and the placeholder indicating to the user how they should name this project compatible with GitHub standards. And beneath that, let's go ahead and simply handle any errors using the field error and the prop errors, field state meta errors. Great. Now, outside of that form field, let's go ahead and duplicate that and paste it here. So this one will be used for visibility. Alright. In this case we don't need the isInvalid. We can immediately go ahead and return. So we can remove it. Actually I mean we can keep it. It doesn't really matter. Not too sure because this is a different component. Sorry. I am going to remove it actually. So the field label will simply say visibility. And the prop here is not going to be an input, so we can get rid of that. And the error too. The prop will be select. So inside of this select, let's go ahead and give it value, field state value. Let's give it on value change, which accepts value, which is either public or private. and it calls field handle change and passes in the value. Then in here let's go ahead and do normal select composition select trigger with the id it needs and select value with a placeholder select visibility. Beneath the select trigger we're going to render the select content with its select items one for private and one for public. Make sure the value has the exact same value as you've defined everywhere else, public, private, in lowercase. So this needs to match what you've defined in your project's schema. Here it is, a casting, public or private. All right. And then the last item that we need is the description item. For that, again, you can copy the first one, repository name. I'm just going to go ahead and add it here. change it to description this field can stay the same change the field label to be description and instead of using the input we render the text area. ID is field name, name stays the same value stays the same, on blur stays the same on change area invalid, the only thing we're going to change is the placeholder as a short description of the project and rows to two. The error rendering stays the same. What's left to do is the submit button. So outside of the last form.field, render a form.subscribe with the usual selector of can submit and is submitting. We can access those fields through a syntax like this and then simply render the button inside type of submit size small class name with full and disabled if you cannot submit or if you are submitting and if you are show a different label creating and the default label create repository great then let's go ahead and create a function called get status icon. So depending on the current status of the export, we are going to display different icons. For export status exporting, it's going to be an animated loader icon. In case it's completed, it's going to be a check check icon with a specific emerald color. If it's failed, it's going to be x circle icon with a specific red color. Otherwise, it's going to be a regular FA GitHub. All of them use the same size. Great. And now that we have that, there is only one more thing left to do. So delete the entire dialog here. And let's do a very simple popover composition open and on open change popover trigger as child let's go ahead and do the following so uh the reason we are going to do this so we're going to create a div here with this class name flex item center gap 1.5 height full px3 cursor pointer text mute foreground border L hover bgx in 30 and inside we're going to render get status icon and a span class name text small of export the reason we have this super specific class name is because this is that button so I was just copying the styles of the tabs that we have right that what I was doing right here that why we have this super specific class name So yes now when we render this we going to replace this old dummy export button which currently does nothing. And then outside of the popover trigger, all we ought to do is render the popover content with class name v80 align start and executes the actual render content. Perfect. We are ready to wire this up. First component we're going to add is the import dialog. For that, we're going to go inside of source, features, projects, components, and we're going to go inside of projects view. Let's go ahead and add an import for import GitHub dialog. We've developed it in the same folder, So we can use a very short path here. Then let's go ahead and add a state here just beneath the command dialog. Import dialog open, set import dialog open. Then in the use effect here, so far we only checked for a key of letter K. Now we're also going to check for letter I, which will import the dialog, which will open the import dialog. My apologies. Okay, make sure that we actually have the event listener. Let's make sure we actually close it. Or maybe... Okay, no, I think this is all just fine. Okay. And now we have to render the import GitHub dialog. We can do it just beneath the command dialog with its equivalent open, import dialog open. and on open change set import dialog open and while we can now open it with a shortcut let's actually give this other empty button with fa github uh an on click right set import dialog open so right now if you go right here and click on import it should show you import from github enter a GitHub repository URL to import. A new project will be created with the repository contents. Amazing. And if you try something stupid, you will see you get a please enter a valid URL. All right. So I will try this out, but I suggest that we try it out together. Let's just finish wiring up the UI components. So one more place we have to visit is the source, features, projects, components, project ID, view. Let's go ahead and import the export popover, which we've just created. It's right here in the same folder. And now let's go ahead and find the placeholder div that we have. Let me find it. Here it is. So after it, find the tab with the label preview. and then in here you will find this div class name flex one justify n so that's good but this this is just a mock function so go ahead and now you do you notice the class name the class name is exactly what we've added here right it's the same class name so we can now delete this and just render export popover with project ID. So let's quickly go into a random project just to see if we can now open that popover, which gives us pre-filled repository name, an option to change the visibility and a description. Perfect. Now I'm going to go ahead and prepare a few repositories for us to test if there are any bugs. So I'm going to start with a random repository I have. this is a private repository so I have to be logged in to try this so you can see the URL is github.com my name and then the repository name and I would suggest removing forward 3 tree forward main so you just have the repository name here and let's click import and of course this is going to fail now the reason it's failing is because I don't even have ingest running, my apologies so let me go ahead and update if I need to and then we're going to see exactly if it works or doesn't so I'm not sure I think I have to import once again there we go importing repository and you can see it's already creating files so this is actually all working I'm super impressed that we did this from the first try and you can see the status is importing and there we go That's it. It was so fast and it worked so well that I'm in AVE and it's completed. There we go. So the first thing it did is it cleaned the project. Then it fetched the repository. It created the folders, public source and source components. And then it created the files and set the completed status and the project is finished. And in the preview here, I think we should be able to also preview it. I think it's just a simple landing screen that I've generated with AI, actually. One thing we didn't try is a binary file. So that's something that I'm yet to try. I'm just going to create a random repository, or I'm going to attempt to fetch some public repository. All right, so I just waited so this installs. Yeah, you can see we can even preview it. And while we're here, we can try exporting this. or if you want to have some more fun go ahead and create a brand new one and simply you know create a simple white plus react to do app something like that wait for it to be created and then we're going to try and export it now that this project has been completed with ai and i have a simple to do here let's try exporting so you can see that the name is already pre-filled with my random project name. I'm going to set it to private and let's set the description to be test description. And let's click create repository. So export has started and you can see that I have a cancel button if I ever want to stop it. But let's take a look at what's actually happening here. So we're getting the GitHub user. We are creating repository. We are waiting for repository to initiate. You can see it took a few attempts. Then we get the initial commit. We fetch project files we create blobs create tree create commits update branch set completed status and that's it we successfully exported to github let's view this on github and here it is the exact file and you can see initial commit from polaris the only thing i'm not seeing is the readme perhaps it still needs to be synced or maybe we made some mistake we will see but that's honestly the least important part of this entire thing. It's that the files are actually in it. Perfect. So the only thing left to check is what's up with binary files. So what I've prepared is I've just uploaded a random image to one of my repositories here. I suggest you do that as well. You can use upload files and just add an image. Make sure it's JPEG, PNG. Basically, make sure it's not an SVG file because that's text, right? Make sure it's JPEG or PNG or some other binary that you have. So I'm just using a simple COVID Antonio icon here, right? And what I'm going to do is I'm going to copy the URL of this repository, which has that, and I'm just going to go ahead and import it here. So I'm going to paste it and I'm going to click import. So let's see what will happen. Will it succeed with that or not. Already I can see that there is something here and when we click on it, we correctly see to do implement binary preview because we are not yet rendering this in any way. I'm more interested in what's here. So it looks like this was successful. It successfully created files. But let's take a look at the convex. So inside of my data here, I have files and I think that so far we shouldn't have a single file with a storage id except one which is called images.jpg and if i go inside of the actual files this is representing your storage you can see that inside i actually have one file and if i go ahead and click on download here uh i'm not sure what this is. I think this might be some mistake because this does not look like the file I have added, but maybe it is. Maybe it's not. I'm not exactly sure. I think it's because of this incorrect content type. I think something's wrong with the extension, but looks like it was uploaded. What I want to try now is try exporting it. So test file one, two, three or test binary. so I'm just exporting the exact same repository now which has images.jpg so I can see if the binary file was transferred perhaps there is some bug happening here we'll leave that to the next chapter don't worry it's already been two hours but I just want to see if we're doing a mistake or not. All right, so I can now view the repository and I have it here and okay, so it's perfectly fine. The image was successfully uploaded, right? So you can see that this is now test binary 1 to 3 and whatever the image was in this repository where I've manually added it via upload, it was preserved through the Polaris project, through our file storage, all the way to a new repository. All right. So let me go ahead. I'm not exactly sure why when I open it, it's in this weird format. Okay. When I open it on my laptop, it actually just opens a normal image. I should have just opened it. So everything is perfectly fine. We implemented everything correctly. Obviously, we don't have the actual preview here, but that's easy. We're just going to show an image or if we can't show content, we're going to say editor doesn't support this type of file. Amazing, amazing job. As you can see when you export something you can keep it in this state and it will even persist through refresh I believe But if you want to restart it you can click close And we have a bug Okay. Oh, yes, yes, yes. We moved those files. I forgot about that. So inside of source, app, API, GitHub, export, we have cancel and reset. Open both of them. I think both of them should have errors. we have to give each of them a higher level okay so we fixed that okay easy fix I think we can now try again let me refresh and I probably have to rmrf.next so I clear cache and do npm run dev again and then restart because we just fixed both, right? Reset and cancel. I think it was just cache that was the problem. And if we try close now, there we go. You can see how it entirely resets. Amazing, amazing job. Let's go ahead and review and merge these changes. So chapter 15, I'm first going to do git checkout dash b. 15 github import export, git add, git commit, github import and export, and then git push u origin 15 github import export. Perfect. You can see that now we are on that branch here. And then I'm just going to go ahead on to the Polaris repository. I'm going to open a pull request and let's review it. And here is the summary by CodeRabbit. New features. We added import GitHub repositories directly into projects. We can export projects to GitHub with customizable repository settings. We have real-time status tracking for import and export operations and cancel export functionality with ability to reset export status. So let's take a look at the comments here. So the first comment is for the cleanup function. Right now what we do is we simply load all the files in a project and we run them through a loop to delete them. Same with their storage. But you can see that CodeRabbit reads the convex documentation and it knows that convex enforces a 100,000 operation limit permutation. That is reads and writes combined. So projects with over 50,000 files will fail. All right, so obviously the solution for this will be to implement batch cleanup, as it said. Right now, for our tutorial purposes, this is perfectly fine. But yes, you should be aware that there are limits to convexes mutations. Same with wherever you deploy a normal API route, there are limits to how long it can run something. So perhaps this could be a job for a background job or convex workflows. But for now, this is okay. But you should be aware that there is a limit. I try my best to bring this project as close to production as possible. I think you notice that. That's why you watch these types of videos, right? but I have to compromise here and there. In this case, I didn't create batching. So I hope that when you run this project, if you do it in production, please be aware of that. And of course, fix it. It's a nice challenge for yourself. Great. In here, it's warning us about potentially affected peer dependencies of Octokit. So I'm not really aware of this, but yes, you could run NPM audit to ensure that there are no security issues. I think everything is mostly okay. In here, it says the default values won't update when the project loads, but we've tested this and it does work correctly. So I think it's confused because of our optional chaining here. I think that's what confuses it, because yes, usually this wouldn't update once it loads. But I think ours is already loaded this time. So perhaps we don't even need the optional chaining. Yes. And about this, I will try to get more information in the next chapter if it's something serious, but I'm pretty sure it is not. All right. and in here it's basically telling us that we have a very broad catch here so we could mask any real errors again for tutorial purposes it's okay for production yeah you would probably want something a bit more how do i exactly say well not so broad right because i don't even care why this fail. This can fail because of the branch, which is what we assume, but it can also be a million other things. It can be invalid owner, invalid repo, right? So that's what it's complaining about. It's the fact that the moment this fails, we just assume it's the branch, but it can be other things. So in production, you will probably check, you know, for the type of error and what the error returns and then do something. Same thing here. We do silent error handling. You You can see the catch and just console error. So we don't really know which files have failed to error. We don't really keep track of anything. In here is a good opportunity to actually use Sentry logging for this. This could be very useful. So you can keep track of files that fail. And perhaps you can then extract if only binary files are failing or only a specific extension is failing. This is where Sentry could come in very, very useful. so you can analyze which files cause problems the most. Other than that, great, great job. Let's go ahead and merge this pull request. Let's go ahead, git checkout main, git pull origin main. And there we go. So that marks the end of this chapter. Let's just confirm everything here is merged. Let's go ahead and check our graph. There we go, 14 and then 15. Perfect. And yes, that is all. We've connected GitHub OAuth via Clark. We built complete import system with binary file support. We created background export jobs using Ingest workflow. We implemented real-time status tracking with UI components. And we handle repository creation and Git API operations. Amazing, amazing job. And see you in the next chapter. This is the final feature chapter before deployment. We'll add several polish features that make the app feel complete and production ready. We will allow the user to trigger a new project dialog with prompt input, allowing users to create new projects using natural language. We're going to improve our use mutation hooks with optimistic updates for instant UI feedback, making the app feel snappier and faster. we're going to implement billing, turning this project into a real SaaS, and you will learn how to protect certain premium features. In this example, we're going to protect GitHub's import and export feature, but using that example, you can protect under a premium feature whatever you deem worthy of a premium tier. And then we're just going to create some nice warnings whenever a user tries to open a binary file saying that we do not support the preview of that file. We can still of course have it in our database we just don't support showing it in the code editor and lastly we're going to do some AI element styling and interaction polishing. So let's go ahead and make sure we have npm run dev running, npx convex dev and npx ingest cli latest dev. then let's go ahead and start by creating a convex system mutation so head inside of your convex folder and let's go inside of system.ds and at the bottom here let's go ahead and add it as the last function we're going to create a mutation that will be used to create atomic creation of a project and its initial conversation. So the name will reflect that. Export const create project with conversation. It's going to be a mutation. It will accept some arguments and let's just do a quick preparation of the handler. We already know the handler will have context and arguments and those arguments are going to be as usual the internal key and then project name, conversation title, and owner id. Since this is a system mutation and we don't have authentication here we have to validate the internal key to make sure no malicious actors are trying to access this. Let's store the current date into a variable called now and let's create a project by inserting into the project stable with the project name, owner ID, and updated at to be now. Then let's go ahead and create a conversation by inserting into the conversations table, passing along the recently created project ID, title from the arguments, and repeating the updated at variable. Once we have those two, let's go ahead and return them all together so we can work with them since they are related to one another. So this will be used to very simply create both the project and conversation related to that project from one function. Great. Once we have that, let's go ahead inside of source app API folder. In here, let's create a new folder called projects. And then inside of the projects, let's create create with prompt. And then in here, create a route.ts. Let's go ahead and import everything we need. Zod, next response from next server, out, and then from unique names generator, adjectives, animals, colors, and unique names generator itself. Besides that, we're going to need clients for ingest and for convex. Make sure to import them from their password.\nrespectively. And we are going to need API from convex generated API. And we're going to need to find our default conversation title, which I store in here. Let me show you in features, conversations, constants. All right. Now let's go ahead and define what kind of request this API route will accept using Zod. So it's just going to accept a very simple prompt with a minimum length of one meaning it's required. Let's export a POST request from here and inside of this POST request let's start with validation. So are we currently logged in? Can we extract the user id from clerksout function? If we can't let's throw an error. Then let's go ahead and check if we have an internal key. So we know that the environment trying to access this convex mutation is a verified environment because we have no out for system queries. So let me just double check. Polaris convex internal key is the correct one. Great. If internal key is missing, throw 500 because this is something we should set up on the system. This isn't the user's error. This is truly an internal error if it happens. Let's go ahead and extract the body and let's parse the contents of the body using the request schema Zod object we've created above. Now that we have a safely parsed prompt, we can go ahead and generate a random project name using the unique names generator. So the project name will use unique names generator. I put these three dictionaries simply because this combination and hyphen in three letters, I'm sorry, three words just create fun project names and we are consistent with all the other places where we use this. I think it's in project's view. I think it's exactly the same. Adjectives, animals, colors, separator, adjectives, animals, colors. Yes, it's just being consistent. Now let's go ahead and create both the project and the conversation together, which we can now very easily do by calling a specific convex mutation, right? So we already know we're going to return project ID and conversation ID. so let's await convex mutation api system create project with conversation if you're not getting autocomplete here take a look at your server here for example uh oh i run npx convex whoops i meant npx convex dev so let's go ahead and make sure you have npx convex dev running and you should see convex functions are ready so yes in case you were having an underline here is because the function was not synchronized with convex so just make sure you have npx convex dev running now we have to add the json right the body so that's the internal key the project name the conversation title which we're going to just use the default conversation title and the owner id which is user id great now we have to create the user message basically what did the user prompt for this project. So we're going to call convexMutationAPISystemCreateMessage. We have already used this a couple of times in the project, specifically in API messages, so you definitely should have this in your system.ds in the convex folder. Pass along the internal key, the conversation ID, this message is being stored into the project ID, the role, which we can hard code the user because right now we're just storing the prompt, the natural language the user is using to explain what they want this project to be. And then we immediately have to create the assistance message with a placeholder for the processing status. And we should get the ID of that assistant message. So let's go ahead and prepare that by storing the assistant message ID and calling await convex.mutation. let's go ahead and call api.system.createMessage again just as we did above and let's pass in the internal key the conversation id the project id role hard-coded to assistant content being empty and status being processing right we are about to process the prompt that the user just sent us and the assistant message id will be the proper id because the create message returns the message id so just make sure your does as well great let's go back instead of the assistant message id here we've done that and now what we have to do is we have to trigger the ingest job to process the message, lucky for us, we've already developed that. So all we have to do is call the event message forward slash sent and passing the data. Message ID being assistant message ID, conversation ID, project ID, and message being user's prompt. And once we have that, let's return next response dot json and pass in the project id so what's important here is that you double check that you actually have this event in your project you can see i this is the third time i'm referencing that event id so obviously it's the correct one it's the processing message it's the process message one in here what's important is please just check that this message event is correct. So message ID, conversation ID, project ID, message. Make sure that you don't accidentally misspell something here because there's nothing stopping you, as you can see. You can type whatever you want. It's not going to throw you an error. So be careful, okay? Because the bugs around this might be a little funny, right? Okay. So now that we have that, let's go ahead and implement the UI for this dialogue so we can actually test it out. So we're going to go inside of source features projects. Let's go inside of components and let's create a new file new project dialogue.dsx. Let's go ahead and mark it as use client. Let's go ahead and import use effect use state ky toast and use router from next navigation. Let's import dialog, dialog content, description, header, and title. And then from the specific components, AI element prompt input, which we've added by installing AI elements. I'm not sure if that's actually, I think we run chat CN installation on it, but basically you should already have AI elements. If you don't, it is basically from AI SDK, AI Elements. We already have them in our projects because we have been using it in the chat sidebar. My apologies, conversations sidebar. So let me just scroll up. There we go. You can see we already used AI Elements Conversations and AI Elements Message and Prompt Input. And now we are, again, importing from Prompt Input. So that's where we got that. let's go ahead and prepare the id from generated data model let's create the interface new project dialog props which will accept open and on open change and then in here let's go ahead and export the actual component so this is the usual scenario right almost every dialog popover has the open and on open change that's simply because we are creating an abstraction over the composition that ShatCN has given us, which is always having an open and unopened change prop. So let's go ahead and prepare the router hook so we can easily redirect once the project is created. Let's prepare the user's input value here. And let's go ahead and give them a little submitting state for a nicer user experience. Now we have to develop the actual handle submit method. So that's going to be an asynchronous method and its message or its value, right, its prop will be a type of prompt input message which we have imported as a type here from the prompt input now the handle submit will first check if there is no message.text and if there isn't any it's just going to return then let's go ahead and set is submitting to true and let's open a try and catch block inside of the try block let's go ahead and call await ky.post api projects Create with prompt. So this should match 100% what you've written here. App folder API. Projects create with prompt. Make sure there are no misspellings in this folder name and no misspellings in here. Create with prompt and route.ts is a required file name. So make sure you didn't accidentally misspell any of that. We already know what we are going to extract from here. the project id so we can prepare that and then let's just go ahead and add some body to this post so we're going to be using json and we will pass along the prompt which is going to be message.text and just trim it great and then we can go ahead and actually request json back and to make it type safe we can go ahead and cast a type project id to be a type of id projects and just like that when you hover over project id here it has the correct type and if you actually look in the route dot ts of the create with prompt you can see we do return the project id perfect so now our front end is completely synchronized with the back end after that happens we can go ahead and render a toast success project created we can close the model we can set the input to be empty and finally we can push to projects project id if catch happens it's most likely because of an internal error or some invalid data so let's just say unable to create project and in the finally block set is submitting to false. So regardless if this fails or succeeds, it will be reset. Now let's go ahead and actually do the composition of a dialogue so we can render it. So using the dialogue component which we've imported above, let's give it an open and on open change prop. Then let go ahead and write the dialogue content with show close button being false and class name being on small maximum width is large and padding is zero Then let's go ahead and open a dialog header and I'm just going to go ahead and give it a class name of hidden but it's a good recommendation to still add the title and the description for accessibility. So screen readers can access it. So it's not going to be visible, but screen readers can say what this model is doing. Great. Now outside of the dialog header, let's go ahead and create a prompt input. And let's give it an on submit of handle submit and the class name border none with an exclamation point at the end, marking it as important. Inside, let's go ahead and render prompt input body. and then let's go ahead and render prompt input text area with the placeholder ask Polaris to build. On change set the input to events target value, synchronize the value binding to input and disabled is submitting. And then let's go ahead and open the prompt input footer. Let's go ahead and render the tools and finally the submit button which is going to be disabled if user didn't type anything or if we are in the process of submitting so the user cannot spam. Great, that is the UI component. And now let's wire it up so we can test if it works. So we have to go inside of projects, components, and we have to find the projects view right here. Let's start by adding the import first. So I'm just going to add new project dialog. new project dialog like we usually do and we need to have a state for it so new project dialog set new project dialog use state false let's go ahead and create a little helper function here handle new project open change open set new project dialog open and let's go ahead and add another if here. If event.key is letter J, you can of course modify these hotkeys. Let's go ahead and prevent default and let's set new project dialog to be true. So we can now open it, the hotkey. Perfect. And I'm not sure if we even need handle new project open change. I think this is simple enough. so in the new project dialog here let's add this to be open and on open change set new project dialog open as simple as that great now what we have to do is we have to find a button which is this one which right now just creates a new project so that's actually not going to be the case anymore instead what's going to happen is set new project dialog open will be set to true and I don't think we have to do anything else here I think we can just test it out let me just see if we can now remove this import looks like we can perfect we can remove use create project and the import so let's try it out now I think this should work just fine just make sure you have your ingest running make sure you have the necessary credits let's try a shortcut. There we go. So ask Polaris to build a simple React plus V2Do app. And once you click enter, it should redirect you to that project. And you can see that a new conversation has already been established. The title has been created and the AI is currently thinking and it should start creating the code any moment. And here we go. Just like that, we have kind of improved the user experience so that they don't have to first create a new project they can like immediately prompt it right and it's just a regular to do app great so what we have to do next is improve all of the missing optimistic updates so I think when you search for to do in your app you will see we have a bunch of these optimistic mutation ones most of them instead of use files so let's go inside of use files and see how we can improve that basically this isn't required it will just make the app feel snappier right so let's make sure we have use mutation use query id and api we have all of that and let's now add a little helper function which we're going to need since when we are optimistically updating we have to simulate the same behavior as on the server so we're going to implement a function to sort files folders first then files alphabetically within each group that's the logic we are going to be using so let's go ahead and define sort files and let's go ahead and prepare a type here so t extends type which can be either file or folder name which is a type of string so that's where we're going to have files which are going to be an array of that type and we expect back an array of that type so inside of here let's go ahead and return spread files sort them so you have two files now if first one is a type of folder and the second one is a type of file return minus one which will sort them folders first if first one is a type of file and the second one is a type of folder return one which will do the opposite meaning again sorting folders first and for the rest let's go ahead and do locale compare so we sort alphabetically within groups. Great, so a little helper functions here. So let's start by finding useDeleteFile. So in here, what we can do now in useMutation is call withOptimisticUpdate. And in here, we have localStore and we have the arguments. And then what we're going to do is we're going to get the existing files from local store, get query, API, files, get folder contents. And in here, pass in the project ID, which will be arguments, project ID. Let's see if we have that or not. Looks like it doesn't accept that. so what we have to do is we have to extend use delete file so let's go ahead and extend it by accepting project id and parent id because the problem is in the arguments we only have the id of the file itself so in order to make the optimistic update work we have to find a way to make the hook aware of the project ID and the parent ID. And that's kind of a problem. I'm going to show you why. So make sure parent ID is optional here. And let's now continue developing. So now we know what project ID is, we know what parent ID is, and now we can load the existing files. And now what we're going to do is just check if existing files are not undefined, meaning they have loaded. Let's do local store set query and let's do API files get folder contents project ID, parent ID, existing files dot filter. Let me just fix the type existing files dot filter get the individual file and check if file id matches arguments id does not match arguments id all right so what this is doing is it is simulating what the actual backend function this one delete file is going to do but it is just simulating that right but you can already see that this might not be the greatest of examples simply because we should also kind of hide all its recursive children elements, which isn't too big of a problem because on the front end, I think the UI immediately hides that. But let's actually try it out to see the example and why you would want optimistic update or maybe you prefer not doing it. So the first kind of caveat is that you have to complicate the developer experience. So if you search for use delete file in your project, you will find that you use it inside of tree.tsx. So let's go ahead and add it here. You can see that now I have an error here because I have to extend it with the project ID and the parent ID item.parent ID. So let's see the difference, for example. Let me open one of the existing projects here. And you can see that now when I delete, it is instant. And I mean instant, absolutely instant, faster than the actual backend. So let's see the difference. So let's try, you know, commenting with optimistic update out. It's still going to be fast because convex is fast, right? Let me refresh just in case. But you will see like a very slight delay. See, it's very, very small, but it is visible. whereas with optimistic update is absolutely instant but that's not the only thing so for example let's say that we want to throw an error if whoops if true so let's simulate this let's always throw an error what happens then still optimistic update will immediately delete it but then it should bring it back you can see that's what happens that's the power of optimistic updates. So it gives the user an idea of what was supposed to happen, but it has the ability to roll back if it goes wrong So it up to you if you want this compromise or not I personally think optimistic updates really really make the app appear faster because for the user it almost like there are no network requests right For the user, the moment you hit it, it happens instantly. Now, when it actually happens, might be five seconds from now, right? The user doesn't even know that your app is slow simply because you've done a good job with optimistic updates. But the caveat is that we can see you kind of have a more complicated example here. So if you want to, you can follow along to see me develop the rest of this. So rename file, I'm going to extend it with the exact same props here. And I'm going to go back inside of use files here. And I will add the exact same props here. So project ID and parent ID, which is optional. And I'm going to go ahead and do the exact same extension. local store and arguments. We're going to fetch the existing files. But first, let's just check inside of arguments. Okay, so we only have ID and new name. Sometimes you don't even need to pass these simply because you have them in arguments. But in this case, we do need to pass them. And now what we're doing here is again checking if the existing files have loaded. Since in Convex, the result is never undefined. It's either a result or null. If it's undefined, it means it's loading. So now let's go ahead and go update our files. So we're just going to go ahead and go over existing files. And when we find our file with the ID, let's simply change the name using arguments new name. And for all other files, just return their current state. And once we have the updated files, we can simply add it to the local store using setquery, API files, get folder contents with project ID and parent ID. And very important, you now have to sort files again because you can change the alphabetic order, right? So that's kind of the complexity that we have to do here. Okay, let me see if I'm forgetting to close something. We have this, we have this. am I missing something? let me see, expression is expected let me just check why this is throwing I'm probably missing something well definitely missing something, let's see use mutation API file is a renamed file with optimistic update this seems to end correctly and then we open this that seems to end correctly too local store get query api dot files dot get folder contents is that maybe problematic I don't think it is that seems to end as well then we have this we have this dot map let me see dot map might be problematic are we maybe missing something here? No. I'm trying to figure out what's the error. Let me see if the error is even in this file. It is definitely okay. Oh, I should not add a semicolon there. Great, so now the rename is also instant, right? So if I change this to instant, you can see it's immediately renamed. If I go ahead and change it to alphabetically, you can see it's immediately sorted to the top. Much faster than if you were to wait for a request to happen. So that's the gist of optimistic updates. They basically make your app appear faster than they actually are. So a few more places to do this. Delete, let's see, create file. Yes, so project ID and parent ID. Save that. Let's go inside of use create file. Let's go ahead and prepare the props here. Project ID and parent ID. And let's go ahead and prepare the with optimistic mutation. Okay. Let's check what we have in the arguments. So for example, in here, we have the project ID in the arguments. So we actually don't need to pass it here, which kind of simplifies things on this end, but we still don't have the parent ID. Yes, as I said, I mean, the developer experience is a little bit worse, but it might be worth it. And then here for the project ID, you are just going to use arguments.projectID because you can. Okay. And now that we have the existing files here, let's go ahead and again, check if the existing files have actually loaded and we're just going to simulate creating a new file now so let's go ahead and create an object for the new file let's go ahead and mock a random ID let's mock random creation time project ID. Oh, do we have parent ID in the arguments? Let me see. Oh, we also have parent ID. Great. So we don't need any of them in the create file then. My bad. Yeah, some of them work just the way they do. So arguments, parent ID. Great. That's even better. So parent ID. Let's go ahead and add name and content. Let's make the type B file updated at now. And let's go ahead and fix this error by adding slint disable next line react hooks purity and say optimistic update callback runs on mutation not on render to explain why this is okay great and then let's just go ahead and update the local store with the new file so local store set query project id arguments project id parent id arguments parent id and then make sure to sort files and just append the new file here. So then again, you will see this works instantly too. If I go ahead here, new file, something TS instantly added, right? There's no question about it. It works super, super fast. Great. So that's it for the use create file. And it is almost identical to use create folder. let me see inside of use files if I scroll down here create folder we have the parent ID and we have the project ID so we don't have to extend this at all in fact I'm pretty sure we can just copy the entire with optimistic update here and chain it here like so we're just going to make it a little bit different. So the existing files, all good. This is all good. Instead of new file, this should be called new folder, just so we stay true to what we're actually developing. The type should be folder. There is no content. And this should be new folder. So again, the exact same behavior now if i go ahead and create a new folder super fast immediately created right and again uh the same is true if i for example go inside of files let me see create file if i go here and if i decide to throw an error every single time you can see what happens so new file.ts immediately it gets created and then it's reverted right so optimistic update from convex handles all of that it really makes the app feel just that much faster all right and here's what i would give you as a challenge now uh i mean if you want to we can uh you can just watch me do it but try and do it for project creation right if you want to you can see it takes like a second before it's created here uh but it's not terribly important for it to exist here simply because you can see that even when it's created, we actually get redirected there, right? So it's not really important that the user instantly sees it in here. So we would implement this inside of conversations, hooks, use conversation. It would be use create conversation. And let's see. So you just need to use the argument project ID. So if you want to pause the screen and try and implement optimistic mutation for use create conversation and then I'm going to show you the result. Okay, so for those of you who want to see the result, this is it. So we will start by loading the existing conversations. We will map arguments, project ID here, the date now we can just copy whatever excuse we had from the use files, right? So we get rid of that like so and we are just creating a new conversation object here and again map the project id as arguments project id and you are updating the query get by project and don't feel discouraged if you didn't manage to get this yourself the truth is this would be way easier if we added these optimistic updates when we developed these hooks. Because right now it's kind of hard to recall why am I setting query in GetByProject? How are you supposed to know that? I now realize it's probably not that clear to you why we're doing this. Yes, obviously I have access to the original source code so I can see how it's going to look like. But yes, I just hope you understood that you basically have to create a new conversation and then set it to the local query in a very specific cache. In this case, it is get by project cache. That's the one we want. So actually, what I showed you before was incorrect. This is the scenario. When you click on plus, it will immediately kind of appear here. As I said, the optimistic mutation is not required everywhere, right? In some places, it makes no sense to have it. For example here I don think anyone will really see the benefit But in the files one in the file explorer it really makes sense because it makes the app feel that much faster all right so let me see uh if we have anything here looks like we have another optimistic mutation here's an example of where we really don't need it update project settings really no need right i think actually the only ones that made sense here were use files even this last one in use conversation which i told you to try and do yourself in my opinion you can even decide to not do it I don't think there's much benefit to it it's just a good exercise but not much sense in my opinion alright so what I want to do now is I want to try and find that project I had which featured an image so right now I have this very kind of ugly to do implement binary preview so how about we just implement a nice error screen or a nice warning so I'm just going to close everything here and I'm going to find editor view inside of features, editor components, editor view and here it is, isActiveFileBinary so what we're going to do is something much nicer, let's go ahead and start with a container so a div class name size full flex items center and justify center then inside of here another div with flex flex column items center gap 2.5 maximum width of medium and text center then let's render an alert triangle icon from lucid react with size 10 and text yellow 500 like this and let's go ahead then display a paragraph below, whoops, so below the triangle, a paragraph with text small, which will very simply say the file is not displayed in the text editor because it is either binary or uses an unsupported text encoding like this. So just some warning to the user like, hey, this is why that is happening. You can, of course, tweak this to make it look better, especially on smaller devices. Maybe even make the triangle smaller, larger, however you prefer. Great. Now it's time to add premium features. Right now, this export functionality and the import functionality work just fine. So what I want to do is I want to protect them. So head to your clerk's dashboard and click on the billing tab. and let's click on get started. So in here, let's go ahead and click on enable user billing and see if we even have that available. And let's click save. Okay, so if this wasn't available for you, there's a chance it is because you don't allow, let me see, user authentication, you don't allow sign up and sign in with email. So in order for billing to work, you need to allow signing in with email, you need to allow the require email address so just look at my settings and make sure you have them like that i mean the clerk will tell you that when you try to enable billing great so once this is enabled we have to create some subscription plans they already created one for you which is the free tier and now we're just going to create a new one and we're just going to call this pro and let the key be pro. In the description, we're going to say this is a pro plan unlocking premium features of Polaris. Monthly base fee, let's go ahead and set it to like $29.99. You can automatically create an annual discount. Oops, that is not the price I intended. You can automatically set a annual discount. So for example, if someone wants to pay a year in advance, you can go ahead and give them a different deal, right? You can also enable free trial if you want to as well. So let's go ahead and save that. You don't really need to add features for our use case, but if you want to, you could specify exactly what plan has what features. And once you have at least one plan, it's important that you remember the key for this plan, which in our case is pro. So for example, let's go ahead and try something now. Once we have that enabled, let's go inside of import route.ts. And in here, so far, what we do is we just check for the user ID. But you can also extract has from here. So now let's go ahead and first check if we have the user ID. And then let's do has pro to be has plan pro. This is the key that's important. And if the user doesn't have pro, return next response dot JSON error pro plan required with a status 403 as simple as that so i'm gonna go ahead and try do that now so i'm gonna i think it might be a good idea to just restart your app simply because we just enabled something in clerk so just to make sure it still works so let's restart let's go ahead and try and clone something. GitHub.com, code with Antonio, Polaris. Let's click import. And you can see we have an error. Unable to import repository. And you can see the result is 403. That is because I should not be able to do that. I don't have the pro plan, right? So now let me just go ahead and also add this to one more feature so we can wrap up our backend coding. Let's just add it to export, for example. So after we check for that, check for has and extract has from await out. Make sure you add it. So as simple as that. And make sure you throw an error called pro plan required, because again, we're going to use that to check on the front end if we should tell to the user a specific thing. So now let's go inside of the import GitHub dialog component. And in here, we can catch errors, right? So we already catch if the error says GitHub not connected. But now let's go ahead and do pro plan required like this. So if body error includes pro plan required, and the body doesn't need the optional chain method, let's throw toast error upgrade to import repositories. and let's add an action label upgrade on click open user profile like so and let's go ahead and call on open change set to false and let's do an early return so now we don't need to do this in an else if so let's try it out let's see what happens if I try import now you can see it says upgrade to import repositories and then I have to go inside of what I now have which is billing and in here I have to switch plans so for example if I want to do monthly it's 29 if I enable annually it switches to 10 I can subscribe and I immediately have pay with test card mode right here and just like that payment was successful if your hangs and it doesn't work it could be because you have an error here which will tell you that you don't have proper course set up this is why it was important for us to properly configure the next config to use credential less because if you use require corp i think that's the other one then it will block stripe from being able to work so make sure you put credential less or if it's still not working try deleting the headers entirely or or specifically maybe make them inside of projects like this then it shouldn't matter which one you use because the upgrade one is on this one but still regardless of what you do here i would recommend using credentialist because you never know where the user might get the update prompt great so now this should work let me go ahead and try it. But since this is going to be a long action, I will open localhost 8288. Let's try code with, oops, github.com, code with Antonio Polaris. I'm trying to import this very source code. You can see now it works. And yeah, okay, it's fetching the repo. And I'm just going to cancel it because it's a big repository so it's going to take a while for it to load you can see it's trying to replicate all the folders already uh great but that works uh let's see what's up with export right so exports should also work i'm just going to go ahead and create a random repository exporting to github and i'm not sure if it's going to export anything at all because we only have empty files here but okay it's it's unable to export because there are no real files everything's empty but we didn't get an error right whereas if you try with another account which doesn't have billing which by the way i think you can also control through your users you can you can like shut down their billing and stuff then this action would still show you the prompt that you have to upgrade so it was that easy to turn this into a real sass which is actually connected can be connected to your stripe if you go inside of billing and explore all the other things it can do in here you have the dashboard you can see your monthly recurring revenue your total revenue and your actual users here so it's that easy to enable billing using clerk i think this took less like five minutes to do amazing absolutely amazing uh one thing we forgot to do so the same thing we just did for pro plan required let's copy that and let's go inside of export popover in here and let's do the same thing\nright here. So if pro plan is required, set open to false and return. And we can remove the optional chain for the body. Great. So now even in this export popover, we're going to get the same toast to open user profile and to allow the user to upgrade. Great. So I think the only thing that's left before deployment is improving this conversation. Because if I ask it something like, create me a simple JSX snippet, you will see that this is a very weird color. It's barely visible against this background. And if we get a code snippet back, you're going to see that it just looks bad. So what I want to do is I want to go inside of prompt input, which you can find. Let me go ahead and see. You can find it inside of source app components AI elements prompt input. And in here we're just going to modify some classes. Okay, so I want you to return the markdown in here, not create a file. I'm trying to make it explain me some code in here and not just create me a file. Anyway, let's go ahead and try and do something. Let's go ahead and find a component called input group. Input group. Alright. Input group right now only has overflow hidden let me see if that is the file I'm looking for are there other instances of input group so okay so find this one input group class name overflow hidden and let's give it rounded large and exclamation points all right and that's going to make it a bit more rounded okay that is step one okay and we can finally get some snippets here perfect you can see they look very bad okay now let's go ahead and find ai elements message.tsx again inside of source components ai elements message.tsx i want to find some class names here maximum width so search for maximum width and change 95 to 80 I think it just looks better for our use case. I mean, these are just tweaks. None of this is terribly important, right? And now to make it look better against this background, let's go ahead and scroll down. And in here, you're going to find this big class name for the message content. And let's see. We should see BG secondary here. So group, if is user, it uses BG secondary. I want to change that to BG Accent. And I think that that already should massively improve there. You can see how nicer this looks now just by using BG Accent instead of the other one. And I think what's left is for us to fix the message response. Let's see. So scroll down and find message response. There's a lot of files here. Okay, here it is, message response. And in here, in the stream down, let's add the theme to be one dark pro and one light as the alternative. That immediately, you can see, fixes the look of the markdown. And in here, I do want to add some classes here. so let's do a target to inner div and use bg accent and then another target to inner div and rounded medium there we go so just a slight modification it makes this that much more readable right there we go uh i don't think there's a need to edit anything else but yeah this is the place where you can basically tweak any of those things. Perfect. So what I want to do now is just get a few more things ready for deployment. For example, you can see we just have a bunch of errors in this file right here, and there are probably some other files where we have errors. And kind of the easiest way to test that out is by running npm run build, because this is what all of the deployment services we're going to use are going to run. So let's see, can we locally build something? This will now probably fail because we have some unresolved files. So let's see which files they are and if we can fix them. And looks like the first problem is actually in our layout here, which is interesting because CodeRabbit actually warned us about this. Let's visit the layout files that we have. Oh, looks like we only have one project ID. I think the problem is that we are defining the project ID to be an ID of projects when it's actually a strings and instead we should cast it as the ID I think just by doing that change making sure that our params are not defined in funny ways that should fix it but now I'm worried I think that we did this a lot of times so let's see params promise we also do it in a page projects project id so open that page app projects project id page and let's go ahead and do the same thing here uh we're going to change this to be a string and then cast it as id projects so this way we won't have any problems with build so let's go ahead and try build again and see what other files are failing. All right, so it looks like we have, as expected, instead of AI elements, we have some unused ts expect error directive. I was searching here if we can perhaps use lint instead of build, but I found it doesn't yield the same results. So let me see, confirmation.tsx inside of AI elements here okay ds expect error so if I remove it it still has some errors here so let's get rid of them some more basically just removing everything until it's satisfied if you don't want to do this there also is a solution for that you can visit nextconfig.ts and inside of here, you can open TypeScript, and you can add ignore build errors and set it to true, and this way you won't have to fix your build errors in order to build. But if you want to go along and fix this, let's go ahead and run npm run build until everything works. I assume it mostly going to be fixing the AI elements files So as I expected some more TS expect error So maybe we can actually search through our code base, well, specifically AI elements, and click find in folder and search through that. Okay, looks like those were the last one in tool.tsx in the AI elements folder. So this comment, let's just remove it wherever it is. And that seems to work. And I think while we are here, we can just focus on the AI elements folder. And let's just try going over each file here and see if any of these turn red. So we know that there is a lint error inside of them. For example, inline citation of mine has an error here. And I'm not going to fix the way it works. I will simply disable lint for this line, for example, simply because I didn't write these components. These were chatty and added. So I don't want to accidentally mess up the way they work. All right. And that's kind of the way you can fix all of these components. Of course, for example, prompt input has some problems. Let's scroll down a bunch of annies. So I'm going to quick fix and I will disable my explicit any for the entire file. Okay. And then I'm going to scroll down again calling set state quick fix and i'm going to disable that for the entire file as well looks like we still have some errors so let's scroll up here to find it again i will click quick fix and disable that rule for the entire file like that okay prompt input queue reasoning again some problems in the reasoning i'm just going to go ahead and disable that rule right so i mean in production obviously you could take more care of these components but right now we just care about being able to deploy and i want to show you how you can go through these files and just add these eslin disables so they allow you to properly build your app so that you can go ahead and focus on other things and it looks like that's it great so all the chat cna elements files are now fixed, but we are still not fully ready because we still have components UI. Any of these could be problematic too. So now I'm going through them just to maybe be ahead of npm run build if it fails. And the fix is exactly the same. Or if you really notice that you're not using a specific component, you can also remove it. Just make sure there are no other components which use it as a dependency. Alright, looking good so far. No errors. Item, keyboard, label, menu bar. We really do have a lot of components. Obviously you can get rid of those you're not using, but I always like to add all of them. It's easier to work that way. Sidebar, skeleton. Most of these look very good. trying to find any that turns red how about resizable okay resizable seems to be problematic it's exactly the one this caught and I think this is actually a problem in the version here I think I saw it in github so I'm going to search if we actually use resizable anywhere we don't because we use allotment panels we can just remove resizable in that case. Let's try npm run build again. Looks like we have some problems in components navbar use rename project. Let's see what that's about. That's a component we haven't worked in in a while. So it is inside of projects components navbar. And looks like when it comes to use rename project, we were passing project ID, but we later decided we don't need to use project ID. It looks like everything works just well without it. So again, npm run build until it works. It's not a nice process but yeah. And sometimes it's different results on Vercel and on other, you know, wherever you deploy it than it's in your local one. So yeah, it's really fun to do. But, you know, this is just to make sure you don't have any build breaking problems in your app. Most of this, you know, the app would work just fine. It's just that for safety reasons, type errors are very important. And there we go. So this is how it looks like when it all goes well. Finished TypeScript and you will see your app. Amazing. So let's go ahead and merge all of those final changes. So chapter 16, billing and final polish, git add, git commit, 16. billing and final polish and git push u origin 16 billing and final polish. Then let's go ahead and open a pull request like we usually do. So compare and pull request and let's see the changes with it. And here we have the summary of the last chapter. We created projects with AI-powered prompt descriptions. We added pro plan requirements check for GitHub import and export features. We enhanced message display with improved styling and syntax highlighting. We improved binary file handling with informative messaging. We optimized file operation with instant local feedback that's referring to optimistic mutations. And we refined project creation workflow with the new dialog interface. Amazing. We have two comments. In the export popover, in the toast error, while I do tell the user to upgrade, I accidentally copied from the import one. This should say upgrade to export repositories. So it's a minor issue, but yes, it's a mistake. And in here, we used class name hidden where we should have used screen reader only. or a component visually hidden to display this. So yes, go ahead and fix those two mistakes if you want. Other than that, let's go ahead and merge this pull request. Amazing, amazing job. Let me go ahead and go back to the main branch, git pull origin main, which will now synchronize those changes. There we go. We can now build our project. And as always, I like to confirm with my graph here that everything's fine. so 15 and now 16 amazing so the only thing left is to deploy let's go ahead and see what we've done here we created new project dialogue with prompt input we implemented optimistic updates for instant ui feedback we added pro plan billing gates for premium features created binary file preview warnings and polished all the ai elements amazing amazing job and see you in the next chapter in this chapter we're going to go ahead and deploy our project we're going to start by creating a new project on Vercel configuring Convex configuring Ingest and finally testing out if everything works so using the link on the screen you can visit Vercel or simply visit Vercel.com and I would recommend creating an account or logging in with your github so all of your repositories are automatically synchronized once you log in you will see a dashboard similar like this and go ahead and click add new project In here you will see a list of your repositories and here is our repository Polaris. Let's go ahead and click import. Now in here, the framework preset is already recognized, but we're going to have to do some modifications to the build command and to the environment variables. So for the environment variables, the first thing we're going to do is simply visit our environment variables right here. Let's just copy all of them. It's always the easiest thing to do. And you can just paste them. And all of these will be added. Convex deployment, next public's convex URL, Polaris, convex internal key. Every single one of these here is going to be added. But that's not all we need. So if you look at convex's documentation and head into production, Vercel, in here they have the Vercel marketplace integration but since we already created our own convex team we are better off following these instructions so just scroll a bit down basically until you see connect your convex project to Vercel so we're just doing this right now and what we have to do is we have to override the build command so this is the command that's going to have to be the build command. npx convex deploy dash dash cmd npm run build. So let's go ahead and open build and output settings and change the build command instead of next build to be npx convex deploy dash dash command npm run build. All right. Let's see what else we have to do here. So we have to set up convex deploy key environment variable. So let's go inside of our project settings so make sure you're inside of your project here and in here you have your project settings and click generate production deploy key to generate a production deploy key so right now in here you will see that this is a personal development convex right usually what you would do in production is you would switch from development to production right here keep in mind that by doing this you also have to add all of the environment variables and this is basically a completely blank slate right so just because it's easier right now you can stay in the development instance and in here you have all of your environment variables if you want to you know for real production cases later on you would obviously want all new environment variables you wouldn't actually reuse any of these all of these should be local right but if you want to they offer you an easy copy all which you can then add in your production but for now let's stay in development now let's go ahead inside of url and deploy key click show development credentials and let's click on generate development deploy key so the name for the deploy key i'm just going to call this polaris let's go ahead and click save and here it is this is our key so just go ahead and copy it and let's see under what key do we store it so we store it under convex deploy key so let's add one more convex deploy key and let's paste that in here all right so that's it for the convex we are also have to set up ingest but it's easier to do that after we deploy so for now let's just hit deploy with all of these things here and we're just going to see if it fails if something goes wrong. So this is why we tested the build command locally. So we don't have to watch it fail here, but still it's possible to fail. So let's just see if it succeeds or not. And as we can see, my project was successfully deployed, though we need, we're not ready yet, right? So what we have to do now is we have to visit ingest and we have to create an account because so far we've only been using it in general, right? So I'm going to go ahead and log in. Once you create an account, I would recommend clicking down here and clicking on switch organization and click on create new organization. This way, everything you do for this project is isolated. So I'm going to call this Polaris and click create organization. And let me go ahead and just try Polaris and some unique slug. There we go. now we have a new organization and what we're going to have to do now is we're going to have to integrate ingest with Vercel so click on integrations and find Vercel and click connect so let's go ahead and let me just zoom out a bit so I can see there we go so click connect Vercel to ingest and this will open redirect to Vercel click add integration here if you want to You can add it to all projects or you can choose specific projects. All right. So make sure to select your organization where you have it here and then go ahead and continue. So if you used Ingest with Vercel previously, there is a chance you're not seeing your project here. For example, right now I cannot see my project. I even tried creating a completely new account on Ingest to see if it was because of that. I tried clicking this again and again. You can see I already have it installed here, right? But it still didn't load my Polaris project. You can see it only loads my old project. So I found out that if I go inside of Vercel and click on integrations in here, I can now see ingest and I can go ahead and click manage, manage access. And now I can select Polaris from here. I can even remove my old project or I can simply choose all projects. But for example, let me just add Polaris and click save. And now that I have that, I'm going to go ahead and try refreshing this a few more times to see if now it's going to appear. So I'm going to connect to Vercel again. Click here where it's already installed. And we're going to see, there we go. Now it works. So let me go ahead and click Polaris and save configuration. so now this will automatically add ingest signing key and ingest event key to our environment variables so you can see that polaris is now enabled let me just click configure here to see what else we have to do and i think that we can also take a look inside of our versell let me try and find my projects here is polaris and in polaris here Let me try and find the environment variables. Just a second. They've changed their sidebar, so I'm confused a bit. Switch to old navigation. Let me try and find... Okay, so here, settings, I can find environment variables. And you can see we have ingest event key and ingest signing key. All right. I mean we were just confirming whether they are there or not and in here we have a warning where cell deployment protection might block syncing use the deployment protection key option below to bypass so let's go ahead and try and learn more about that because I think this will improve the way this works so basically just click on your project here right click on Polaris let me go ahead and see how we can find this app back OK No syncs found I think once these two have been added we also have to wait until the deployment finishes I forgot about that, yes. This also automatically triggered a rebuild of your app. So whether you are on the new navigation or old navigation, simply find your deployments, and you should probably see another deployment happening here, like this. And once that is finished, there we go so successful again and now it should be found there we go so something is happening but it is failing probably because of that thing uh it warned us about that's uh vercel might be blocking it so we have to fix that protection url i'm just trying to find my way back here. So you can always go to integrations, click on Vercel, manage, and in here, there we go, you have Polaris project. So let's click configure here again, and let's see what's up now. So okay, still that warning. So I'm going to click learn more under the deployment protection key here. And let's go ahead and see what we have to do. So we shouldn't do this, we should configure protection bypass keep in mind that this may or may not be available depending on the pricing plan but I think it should be available so to enable this you will need to leverage Vercel's protection bypass for automation feature so let's go ahead and enable it protect bypass for automation so we're going to go into Vercel here make sure to select your project let's go into settings and what was the name of that feature protection bypass for automation here it is let's go ahead and click add and i'm just going to give this a name ingest and i will leave this blank so it generates a secret great then i'm going to go ahead and copy that secret here and i will add it here and click save configuration like that and that should fix this problem with uh versell blocking the sync i think we just have to wait a second to see if it starts to work api ingest is correct and okay let's just go back and see if maybe this will now start to work at this point we can already visit our app to see if something else might be broken so when it comes to your app's domain make sure to use this one there's a difference between a deployment url and your actual domain for the project so let's go ahead and start with this one as expected we are unauthorized and now we have to log in now we're just gonna go ahead and log in with github here and looks like i have my project here so let me go ahead and create a new one and i will try and do a simple React plus VIT to do app. So this will be interesting because we have to see whether it will actually start doing this or not. So let me check my runs here. Looks like no runs are happening right now. So perhaps it was not yet synchronized. So let's see. Checking the app help. so I'm going to go ahead and I will copy my app URL here paste it here API ingest and I will click check no issues found so it looks like it can successfully connect to it and sometimes you just have to wait a bit until it starts to work let me go ahead and find some details here Polaris, perhaps yours is already working. Maybe it's just mine that is stuck. So, okay. I'm trying to find maybe it's in a different environment. Yes, this is still not working just yet. So I'm going to go back into my integrations here. I will go into Rercel and I will go back into Polaris to see what's going on. okay maybe i just have to restart it somehow so let me try some things and see what works so what i did just now is did another deployment perhaps after we enable that deployment protection we have to redeploy so we're going to see if that maybe fixes it or helps in just recognize the app because still if I go into integrations here and select my Vercel I think I still get that warning here as if the deployment protection didn't work I think once this is successfully synchronized it should work so we're going to see if it changes after the deployment and in case you don't know how to do a redeployment you can just click on the last deployment that worked and click redeploy and looks like redeployment was the fix you can see that now in my apps i can find polaris so that's what i forgot to do uh they in fact instruct you to do that i just didn't read thoroughly uh so before syncing with ingest ensure that the latest version of your code is live on your platform right and even in here if you go inside of protection bypass for automation they tell you that they actually add an environment variable, which is a clear indicator that we should have redeployed. And you can see that now I have a synchronized Polaris app and all of these functions right here. So let's go ahead and prepare the runs screen. Let me start a new app, create a Vite plus react to do app. Let's go ahead and run this. And let's see if that will trigger a run or not. There we go. You can see the run is now happening right here and hopefully we will be able to see some project. You can see that the conversation has just been renamed so the ingest is definitely working in production. We just have to see if the files and the communication with Convex is working. And here we have a real-time preview and a finished app in production. Amazing. Everything works great. If there is one thing that's on my mind, perhaps it would be when we created the GitHub OAuth app. We created Polaris, I think. And the homepage URL is set to localhost 3000. So perhaps you might want to change it to your new apps URL. But looks like that's not causing any problems. I think this is what's important, but this is completely independent because you get this from clerk not from wherever our app is deployed that's like a very cool thing about clerk great so yes in case github is causing you some problems try changing the homepage url to wherever your app is deployed and click update application maybe redeploy amazing I believe that marks the end of this chapter we configured the project on our cell configured convex configured ingest and tested out the deployment amazing amazing job thank you so much for being with me through 17 chapters of this very long tutorial and see you in the next one",
  "transcript_chars": 324959,
  "transcript_filled_at": "2026-06-06T15:34:26.022461+00:00",
  "transcript_filled_by": "tk-bulk-groq-retry-20260606"
}