PROD DEPLOY 3 6/11 #2051
Merged
PROD DEPLOY 3 6/11 #2051
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… error in job_poller.py
…bs' into colinmegill/node-delphi
Add reportViz route
* batch debugging * batch polling * only run layer 0 * batch * Improve batch processing with new job type architecture This commit introduces a new approach to batch processing jobs with clearer job types: 1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py 2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py The change fixes a critical issue where jobs were running the wrong script, and adds proper handling of DynamoDB reserved keywords like 'status'. This state machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * narrative batch * create user fix, reset db script, commentsreport list * Unified comment extremity calculation. Stored in DynamoDB. * run delphy python conversion * switch off sh to python * call python job poller * go to create narrative batch rather than narrative batch, explicit * commentsReport linter * tmp citations comment * copy py into docker * poller grab python * wrap json parse for narrative * update delphi for create_narrative_report batch * render all cited comments * make arch ready (#2014) * make arch ready * begin splitting up cdk code * organize cdk * cdk efs and permissions fixes * docker fixes, add block storage * remove default dynamo endpoint * endpoint check fix * use us-east-1 * remove more hardcoded values * adjust delphi dockerfile * fix up setupdynamo func * fix reference * add ssl mode hardcoded * another place to add secure mode * ssl fix * update iam roles * remove hardcoded minio from docker * remove s3 check * change s3 endpoint to none if missingf * more s3 config * fix indentation * s3 logic fix * indentation fix * remove env check * comment out s3 creds * remove location constraint * unique bucket name * remove public read on aws * fix comment * endpoint urls in s3 * remove another dynamo default * more endpoint config * lint and test cleanup * more lint fix * defaults fixes * tsc fix * attempt free up disk space * multi stage builds * more space optimization * more optimization * debug * remove commented out docker code, readme and makefile updates * build fix local vs CI * pass arg to ci * add missing comma * ssl mode fixes * Mark jobs as completed. * Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py * modify defaults and change bucket name * side by side * Improve nested log handling in job poller. * add report bundles to proxy static files... * s3 for delphi routes in node server * batches running * show batch time run job timestamp * runs timestamps * quiet batch check debug logging * fix 700 s3 upload config * hash delim * show timestamp of report in ui * logging * no-op, 600 not used, but changes to topic naming * topic naming logic and display * batch uses new topic naming * clean up topic name quotes * happy memorial day bug squash * no batch limit * llm model as param * concise logging * add topic report route * topicReport shows comments * export report embeddable * export report responsiveness * remove .sh references * bump to 16 gigs default, and numba jit * remove shell script rferences from dockerfile * remove sh commands * bump python deps * run, and show, all visualizations in comments Report * claude code reference * notebooks * python tests against real data, compare with clojure * notebooks * vw analysis output * vw correlation plot * vw output * vw notebook * Revert "vw notebook" This reverts commit 7548529. * biodiversity notebook * pca validation notebook * pca validation notebook fix * add ignore, add analysis check. * gpu test * benchmark pngs * move math/python_conversion to delphi dir * setup docker for delphi * update path naming python_converstion -> delph * corrections to postgres schema * Add PostgreSQL integration tests for biodiversity dataset * Add AWS Lambda function for biodiversity dataset processing * Add optimized math functionality to delphi branch and fix PostgreSQL integration tests * Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath * Cleanup Pakistan workflow and write to DynamoDB. * Add Pakistan test for current implementation. * Add requirements. * umap narrative * move docs. * nuke test gpu direction * add evoc to gitigonre a * merge req txt * delete other req txt * remove notebooks * unified create dynamo tables * add fastapi to reqs * add missing group votes initialization * error handinling in db connect * experimental dockerization * concise readme * HAS TO BE ROOT USER!!! * run_delphi talks to dynamo internally :) * all dependencies in req.txt, and c++ in dockerfile. * run command * let's run the umap pipeline directly, wait on orchestrator * dynamo config for docker * docker endpoint for dynamo config throughout, working * ignore python init * remove enhanced table that was deprecated * add curl to dockerfile * docker compose includes ollama * run delphi script checks for ollama avail, defaults to ll3.18b * ollama generalization vs remote llm, working locally * strip quotes * claude generating report narratives. * ensure datamapplot is in scope — docker fix * run visualization after other steps, copy result out to disk * info, not debug from boto * debug model provider * generate all topics for layer 0 * generate a static plot * ignore png and svg image output from static plots * Abstract docker-compose configuration. * Integrate PCA/K-Means/Rep. pipeline. * 702 color by consesus / divisive * add 702 to run pipeline * merge dynamo and docker take 1 * endpoint dynamo * create table job * little cli * poller * Implement persistent job queue with optimistic locking - Change job table schema to use job_id as primary key - Add optimistic locking with version field - Ensure jobs are never deleted, only updated with new statuses - Implement strong consistency for critical operations - Fix DynamoDB compatibility issues with empty strings * docs for system * deprecated, now unified in top level create tables * add job status cli command * cli checks status checks status * poller logging * rename tables * docs * delphi cdk 1 * docker delphi * add narrative to dockerfile * run delphi with db fix, remove unsued file * stubbing job hierarchy * update job logs claude context * example env for ollama * log missing anthropic key * Add automatic Ollama model setup and environment variable handling for jobs * python error * ignore node modeuls for node scripts in delphi * deprecate delphi compose * parametirize ollama model for topics * complete db rename * fix list convos in cli * remove default model * get rid of default model * service url claude for client report * try catch around conversation.py * hello world move delphi to report client * Load from DynamoDB. * commentsReport route * s3 and submit job from client * no viz error message * Load data properly from DynamoDB and check results are populated. * guard llm names * docs * success showing report in commentsReport * remove cleanup script * add uuid * node batch route * Add PCAPlot component with ASCII-style symbols for high-density retro visualization * don't pop browser * batch reports endpoint * batch reports endpoint * skip static, don't pop browser * don't pop browser * batch debugging * batch polling * only run layer 0 * batch * Improve batch processing with new job type architecture This commit introduces a new approach to batch processing jobs with clearer job types: 1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py 2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py The change fixes a critical issue where jobs were running the wrong script, and adds proper handling of DynamoDB reserved keywords like 'status'. This state machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * narrative batch * create user fix, reset db script, commentsreport list * Unified comment extremity calculation. Stored in DynamoDB. * run delphy python conversion * switch off sh to python * call python job poller * go to create narrative batch rather than narrative batch, explicit * commentsReport linter * tmp citations comment * copy py into docker * poller grab python * wrap json parse for narrative * update delphi for create_narrative_report batch * render all cited comments * make arch ready (#2014) * make arch ready * begin splitting up cdk code * organize cdk * cdk efs and permissions fixes * docker fixes, add block storage * remove default dynamo endpoint * endpoint check fix * use us-east-1 * remove more hardcoded values * adjust delphi dockerfile * fix up setupdynamo func * fix reference * add ssl mode hardcoded * another place to add secure mode * ssl fix * update iam roles * remove hardcoded minio from docker * remove s3 check * change s3 endpoint to none if missingf * more s3 config * fix indentation * s3 logic fix * indentation fix * remove env check * comment out s3 creds * remove location constraint * unique bucket name * remove public read on aws * fix comment * endpoint urls in s3 * remove another dynamo default * more endpoint config * lint and test cleanup * more lint fix * defaults fixes * tsc fix * attempt free up disk space * multi stage builds * more space optimization * more optimization * debug * remove commented out docker code, readme and makefile updates * build fix local vs CI * pass arg to ci * add missing comma * ssl mode fixes * modify defaults and change bucket name * side by side * Mark jobs as completed. * Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py * Improve nested log handling in job poller. * add report bundles to proxy static files... * s3 for delphi routes in node server * batches running * show batch time run job timestamp * runs timestamps * quiet batch check debug logging * fix 700 s3 upload config * hash delim * show timestamp of report in ui * logging * no-op, 600 not used, but changes to topic naming * topic naming logic and display * batch uses new topic naming * clean up topic name quotes * happy memorial day bug squash * no batch limit * llm model as param * concise logging * add topic report route * topicReport shows comments * export report embeddable * export report responsiveness * deps update * viz per layer * comments report gets dropdown * switch scan command for query command * fix sorting * we have two dropdowns we should unify later * begin add GSI for perf improvements (#2029) * begin add GSI for perf improvements * refactor to avoid scan * move config to outside of handler func * replace another table scan * clean up CLI dynamo * fix 801 * cleaner batch jobs * remove another table scan * viz job client logic --------- Co-authored-by: Colin Megill <[email protected]> * actions fix * fix reports gsi query * removing zid from responses, client doesn't consume, running * update docs after checsk * viz zid * batch report rid * fix indentation issues * integrate job id * topicVizReport view * fix reportsByJobId * copy * attempt jest test fixes * add clear runner space to jest tests job * resolve port conflict from pg --------- Co-authored-by: Colin Megill <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Christian Weilbach <[email protected]> Co-authored-by: Bennie Rosas <[email protected]> Co-authored-by: Christian Weilbach <[email protected]>
* use same access key name, simplify boto3 connection * remove duplicates
* bad creds fix * clean up env
* switch to on demand capacity for some tables * job poll updates * poller updates final
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.