The project is composed of several TypeScript modules organized in a monorepo. It makes use of TypeScript Project References to share logic and types between modules. The general structure is mapped below:

flowchart LR
  shared --> server
  shared --> scripts
  shared --> frontend
  shared --> extractors
  shared --> backend

  server --> scripts
  server --> migrations
  server --> extractors
  server --> backend

  extractors --> backend

A UMD module that targets both the browser on the frontend, and bun on the backend. Logic that both the backend and the frontend require should live here.

If you want to write code that is shared between multiple modules only on the server-side, then see server

Contents:

• Types that are shared globally across the entire application
• Database schema types
• Utility functions that are shared globally across the entire application
• Core logic for processing common data structures like SheetContent
• Lispable

This is the main module that interacts with the database. (Most) all application database queries live inside the backend/src/service folder. This module also defines the server-side component of the HTTP APIs. It invokes the extractor module to perform extractions.

API routes are built by adding to the routes/index.ts file. See [Writing new APIs] for more details.

API routes in this module should call into the service folder to manipulate and query the database. A good practice is to keep the API handler functions thin so that our business logic is confined mainly to the service folder.

Contents:

• REST API server-side handling
• WebSocket API server-side handling
• Database interface and main business logic layer

Svelte/TypeScript frontend code. Imports from shared. The structure for file organization here could be redone.

A CLI tool that allows us to define new migrations and to apply them on a database. To create a new migration run just new-migration {migration name}.

Tips for creating migrations:

• Make sure not to import/share types from or to the migrations module. The reason for this is that the types for columns, enums, tables, etc should be "frozen in time" within a migration file. Once the migration file is merged to the main branch of the repository, it should be considered immutable, and no changes should ever be made to it. If a decision needs to be reversed, a new migration to undo the schema changes is desirable.
• A migration has two parts: up and down. The up migration should define all schema changes to bring the schema to a new state, and the down migration should bring the schema back to the state it was before the migration took place. Note that data can be lost in either step, but the schema changes themselves should be reproducable/revertible.

To apply migrations, run APP_ENV=development just migrate.

After writing and applying your migration, run APP_ENV=development just gen-schema to regenerate the schema file

Contains code that is shared across all server-side modules.

Contents:

• Code to set up the connection to the database
• Code to retrieve configuration variables from the environment
• Code that calls out to external APIs

Contains CLI scripts that are mainly just used for development.

Contents:

• Code to update the list of Zip codes
• Code to set up a psql connection to Postgres
• Code to generate the schema file from the database
• Code to ensure that the default field column definitions are up to date

You will need to install bun, just and node (-- TODO: maybe unneeded now?--) on your system.

To use Maptiler maps locally, request a Maptiler API key and store it in frontend/.env.local as VITE_APP_MAPTILER_API_KEY=pIX3QnJrp8jqEgVe3kgc.

• just deps to install all dependencies.
• just install-deps installs dependencies and updates the lockfile - do this after adding a new dependency.
• just watch-frontend to start the frontend with hot module reloading at https://localhost:5173/.
• just watch-backend to start the backend in watch mode at http://localhost:5000/. Optionally prefix with APP_MOCK_FETCHES=false to disable third party API mocking.
• just lint runs on commit, does basic linting. Runs in CI.
• just tsc type-checks all modules except for frontend. Runs in CI.
• just svelte-check type-checks and lints the frontend module. Runs in CI as just svelte-check-ci.
• just test-extractors runs the tests for extractors. Runs in CI.

For the extractor system which needs to make requests to third-party APIs, you can mock the responses so that during development no requests are being made unnecessarily. To do so, run with APP_MOCK_FETCHES=true. (note that the justfile command just watch-backend sets this by default, and must be run like APP_MOCK_FETCHES=false just watch-backend to disable mocking). The VS Code debug configuration also sets MOCK_FETCHES=true by default.

To configure a new endpoint to be mocked, add an entry in mockConfigs in extractors/src/test/mock.ts. A configuration is a Tuple<RegExp, string> that consists of a regular expression to match request URLs and a string referencing a filename relative to the fixtures folder that contains the desired response to mock.

NOTE: future enhancements to the mocking system may include the ability to define functions to determine matching and response behavior.

Credit to Jacob Strieb's clever usage of =IMPORTXML in our original NYC apartment search spreadsheet that inspired this project. Thank you for consulting on all stages of design and ideation.