From e6642514a3031527046ba78a4983598a0b82126b Mon Sep 17 00:00:00 2001 From: James Murdza Date: Wed, 23 Oct 2024 07:43:54 -0600 Subject: [PATCH] chore: update README with new instructions and references to GitWit --- README.md | 262 +++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 188 insertions(+), 74 deletions(-) diff --git a/README.md b/README.md index 0da6e4a..9439914 100644 --- a/README.md +++ b/README.md @@ -1,103 +1,213 @@ -# Sandbox 📦🪄 +# GitWit Sandbox 📦🪄 -Screenshot 2024-05-31 at 8 33 56 AM +GitWit Sandbox Demo -Sandbox is an open-source cloud-based code editing environment with custom AI code autocompletion and real-time collaboration. +Sandbox is an open-source cloud-based code editing environment with custom AI code generation, live preview, real-time collaboration and AI chat. -Check out the [Twitter thread](https://x.com/ishaandey_/status/1796338262002573526) with the demo video! - -Check out this [guide](https://dev.to/jamesmurdza/how-to-setup-ishaan1013sandbox-locally-503p) made by [@jamesmurdza](https://x.com/jamesmurdza) on setting it up locally! +For the latest updates, join our Discord server: [discord.gitwit.dev](https://discord.gitwit.dev/). ## Running Locally -### Frontend +Notes: -Install dependencies +- Double check that whatever you change "SUPERDUPERSECRET" to, it's the same in all config files. +- Right now we are loading project templates from a custom Cloudflare bucket which isn't covered in this guide, but that be updated/fixed very soon. + +### 0. Requirements + +The application uses NodeJS for the backend, NextJS for the frontend and Cloudflare workers for additional backend tasks. + +Needed accounts to set up: + +- [Clerk](https://clerk.com/): Used for user authentication. +- [Liveblocks](https://liveblocks.io/): Used for collaborative editing. +- [E2B](https://e2b.dev/): Used for the terminals and live preview. +- [Cloudflare](https://www.cloudflare.com/): Used for relational data storage (D2) and file storage (R2). + +A quick overview of the tech before we start: The deployment uses a **NextJS** app for the frontend and an **ExpressJS** server on the backend. Presumably that's because NextJS integrates well with Clerk middleware but not with Socket.io. + +### 1. Initial setup + +No surprise in the first step: ```bash -cd frontend -npm install +git clone https://github.com/jamesmurdza/sandbox +cd sandbox ``` -Add the required environment variables in `.env` (example file provided in `.env.example`). You will need to make an account on [Clerk](https://clerk.com/) and [Liveblocks](https://liveblocks.io/) to get API keys. +Run `npm install` in: -Then, run in development mode - -```bash -npm run dev +``` +/frontend +/backend/database +/backend/storage +/backend/server +/backend/ai ``` -### Backend +### 2. Adding Clerk -The backend consists of a primary Express and Socket.io server, and 3 Cloudflare Workers microservices for the D1 database, R2 storage, and Workers AI. The D1 database also contains a [service binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/service-bindings/) to the R2 storage worker. Each open sandbox instantiates a secure Linux sandboxes on E2B, which is used for the terminal and live preview. +Setup the Clerk account. +Get the API keys from Clerk. -You will need to make an account on [E2B](https://e2b.dev/) to get an API key. +Update `/frontend/.env`: -#### Socket.io server - -Install dependencies - -```bash -cd backend/server -npm install +``` +NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY='🔑' +CLERK_SECRET_KEY='🔑' ``` -Add the required environment variables in `.env` (example file provided in `.env.example`) +### 3. Deploying the storage bucket -Project files will be stored in the `projects/` directory. The middleware contains basic authorization logic for connecting to the server. +Go to Cloudflare. +Create and name an R2 storage bucket in the control panel. +Copy the account ID of one domain. -Run in development mode +Update `/backend/storage/src/wrangler.toml`: -```bash -npm run dev +``` +account_id = '🔑' +bucket_name = '🔑' +key = 'SUPERDUPERSECRET' ``` -This directory is dockerized, so feel free to deploy a container on any platform of your choice! I chose not to deploy this project for public access due to costs & safety, but deploying your own for personal use should be no problem. +In the `/backend/storage/src` directory: -#### Cloudflare Workers (Database, Storage, AI) - -Directories: - -- `/backend/database`: D1 database -- `/backend/storage`: R2 storage -- `/backend/ai`: Workers AI - -Install dependencies - -```bash -cd backend/database -npm install - -cd ../storage -npm install - -cd ../ai -npm install ``` - -Read the [documentation](https://developers.cloudflare.com/workers/) to learn more about workers. - -For each directory, add the required environment variables in `wrangler.toml` (example file provided in `wrangler.example.toml`). For the AI worker, you can define any value you want for the `CF_AI_KEY` -- set this in other `.env` files to authorize access. - -Run in development mode - -```bash -npm run dev -``` - -Deploy to Cloudflare with [Wrangler](https://developers.cloudflare.com/workers/wrangler/install-and-update/) - -```bash npx wrangler deploy ``` ---- +### 4. Deploying the database + +Create a database: + +``` +npx wrangler d1 create sandbox-database +``` + +Use the output for the next setp. + +Update `/backend/database/src/wrangler.toml`: + +``` +database_name = '🔑' +database_id = '🔑' +KEY = 'SUPERDUPERSECRET' +STORAGE_WORKER_URL = 'https://storage.🍎.workers.dev' +``` + +In the `/backend/database/src` directory: + +``` +npx wrangler deploy +``` + +### 5. Applying the database schema + +Delete the `/backend/database/drizzle/meta` directory. + +In the `/backend/database/` directory: + +``` +npm run generate +npx wrangler d1 execute sandbox-database --remote --file=./drizzle/0000_🍏_🍐.sql +``` + +### 6. Configuring the server + +Update `/backend/server/.env`: + +``` +DATABASE_WORKER_URL='https://database.🍎.workers.dev' +STORAGE_WORKER_URL='https://storage.🍎.workers.dev' +WORKERS_KEY='SUPERDUPERSECRET' +``` + +### 7. Adding Liveblocks + +Setup the Liveblocks account. + +Update `/frontend/.env`: + +``` +NEXT_PUBLIC_LIVEBLOCKS_PUBLIC_KEY='🔑' +LIVEBLOCKS_SECRET_KEY='🔑' +``` + +### 8. Adding E2B + +Setup the E2B account. + +Update `/backend/server/.env`: + +``` +E2B_API_KEY='🔑' +``` + +### 9. Adding AI code generation + +In the `/backend/ai` directory: + +``` +npx wrangler deploy +``` + +Update `/backend/server/.env`: + +``` +AI_WORKER_URL='https://ai.🍎.workers.dev' +``` + +### 10. Configuring the frontend + +Update `/frontend/.env`: + +``` +NEXT_PUBLIC_DATABASE_WORKER_URL='https://database.🍎.workers.dev' +NEXT_PUBLIC_STORAGE_WORKER_URL='https://storage.🍎.workers.dev' +NEXT_PUBLIC_WORKERS_KEY='SUPERDUPERSECRET' +``` + +### 11. Running the IDE + +Run `npm run dev` simultaneously in: + +``` +/frontend +/backend/server +``` + +## Setting up Deployments + +The steps above do not include steps to setup [Dokku](https://github.com/dokku/dokku), which is required for deployments. + +**Note:** This is completely optional to set up if you just want to run GitWit Sandbox. + +Setting up deployments first requires a separate domain (such as gitwit.app, which we use). + +We then deploy Dokku on a separate server, according to this guide: https://dev.to/jamesmurdza/host-your-own-paas-platform-as-a-service-on-amazon-web-services-3f0d + +The Sandbox platform connects to the Dokku server via SSH, using SSH keys specifically generated for this connection. The SSH key is stored on the Sandbox server, and the following environment variables are set in /backend/server/.env: + +```bash +DOKKU_HOST= +DOKKU_USERNAME= +DOKKU_KEY= +``` + +## Creating Custom Templates + +We're working on a process whereby anyone can contribute a custom template that others can use in the Sandbox environment. The process includes: + +- Creating a [custom E2B Sandbox](https://e2b.dev/docs/sandbox-template) including the template files and dependencies +- Creating a file to specify the run command (e.g. "npm run dev") +- Testing the template with Dokku for deployment + +Please reach out to us [on Discord](https://discord.gitwit.dev/) if you're interested in contributing. ## Contributing -Thanks for your interest in contributing! Review this section before submitting your first pull request. If you need any help, feel free to reach out to [@ishaandey\_](https://x.com/ishaandey_). - -Please prioritize existing issues, but feel free to contribute new issues if you have ideas for a feature or bug that you think would be useful. +Thanks for your interest in contributing! Review this section before submitting your first pull request. If you need any help, feel free contact us [on Discord](https://discord.gitwit.dev/). ### Structure @@ -116,13 +226,13 @@ backend/ └── ai ``` -| Path | Description | -| ------------------ | -------------------------------------------------------------------------- | -| `frontend` | The Next.js application for the frontend. | -| `backend/server` | The Express websocket server. | -| `backend/database` | API for interfacing with the D1 database (SQLite). | +| Path | Description | +| ------------------ | ------------------------------------------------------------ | +| `frontend` | The Next.js application for the frontend. | +| `backend/server` | The Express websocket server. | +| `backend/database` | API for interfacing with the D1 database (SQLite). | | `backend/storage` | API for interfacing with R2 storage. Service-bound to `/backend/database`. | -| `backend/ai` | API for making requests to Workers AI . | +| `backend/ai` | API for making requests to Workers AI . | ### Development @@ -151,11 +261,15 @@ It should be in the form `category(scope or module): message` in your commit mes - `feat / feature`: all changes that introduce completely new code or new features + - `fix`: changes that fix a bug (ideally you will additionally reference an issue if present) + - `refactor`: any code related change that is not a fix nor a feature + - `docs`: changing existing or creating new documentation (i.e. README, docs for usage of a lib or cli usage) + - `chore`: all changes to the repository that do not fit into any of the above categories