Fix comment filtering bug #2111
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tevko
approved these changes
Aug 5, 2025
colinmegill
approved these changes
Aug 5, 2025
tevko
added a commit
that referenced
this pull request
Aug 5, 2025
* Revert "vw notebook" This reverts commit 7548529. * biodiversity notebook * pca validation notebook * pca validation notebook fix * add ignore, add analysis check. * gpu test * benchmark pngs * move math/python_conversion to delphi dir * setup docker for delphi * update path naming python_converstion -> delph * corrections to postgres schema * Add PostgreSQL integration tests for biodiversity dataset * Add AWS Lambda function for biodiversity dataset processing * Add optimized math functionality to delphi branch and fix PostgreSQL integration tests * Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath * Cleanup Pakistan workflow and write to DynamoDB. * Add Pakistan test for current implementation. * Add requirements. * umap narrative * move docs. * nuke test gpu direction * add evoc to gitigonre a * merge req txt * delete other req txt * remove notebooks * unified create dynamo tables * add fastapi to reqs * add missing group votes initialization * error handinling in db connect * experimental dockerization * concise readme * HAS TO BE ROOT USER!!! * run_delphi talks to dynamo internally :) * all dependencies in req.txt, and c++ in dockerfile. * run command * let's run the umap pipeline directly, wait on orchestrator * dynamo config for docker * docker endpoint for dynamo config throughout, working * ignore python init * remove enhanced table that was deprecated * add curl to dockerfile * docker compose includes ollama * run delphi script checks for ollama avail, defaults to ll3.18b * ollama generalization vs remote llm, working locally * strip quotes * claude generating report narratives. * ensure datamapplot is in scope — docker fix * run visualization after other steps, copy result out to disk * info, not debug from boto * debug model provider * generate all topics for layer 0 * generate a static plot * ignore png and svg image output from static plots * Abstract docker-compose configuration. * Integrate PCA/K-Means/Rep. pipeline. * 702 color by consesus / divisive * add 702 to run pipeline * merge dynamo and docker take 1 * endpoint dynamo * create table job * little cli * poller * Implement persistent job queue with optimistic locking - Change job table schema to use job_id as primary key - Add optimistic locking with version field - Ensure jobs are never deleted, only updated with new statuses - Implement strong consistency for critical operations - Fix DynamoDB compatibility issues with empty strings * docs for system * deprecated, now unified in top level create tables * add job status cli command * cli checks status checks status * poller logging * rename tables * docs * delphi cdk 1 * docker delphi * add narrative to dockerfile * run delphi with db fix, remove unsued file * stubbing job hierarchy * update job logs claude context * example env for ollama * log missing anthropic key * Add automatic Ollama model setup and environment variable handling for jobs * python error * ignore node modeuls for node scripts in delphi * deprecate delphi compose * parametirize ollama model for topics * complete db rename * fix list convos in cli * remove default model * get rid of default model * service url claude for client report * try catch around conversation.py * hello world move delphi to report client * Load from DynamoDB. * commentsReport route * s3 and submit job from client * no viz error message * Load data properly from DynamoDB and check results are populated. * guard llm names * docs * success showing report in commentsReport * remove cleanup script * add uuid * node batch route * Add PCAPlot component with ASCII-style symbols for high-density retro visualization * don't pop browser * batch reports endpoint * batch reports endpoint * skip static, don't pop browser * don't pop browser * batch debugging * batch polling * only run layer 0 * batch * Improve batch processing with new job type architecture This commit introduces a new approach to batch processing jobs with clearer job types: 1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py 2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py The change fixes a critical issue where jobs were running the wrong script, and adds proper handling of DynamoDB reserved keywords like 'status'. This state machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * narrative batch * create user fix, reset db script, commentsreport list * Unified comment extremity calculation. Stored in DynamoDB. * run delphy python conversion * switch off sh to python * call python job poller * go to create narrative batch rather than narrative batch, explicit * commentsReport linter * tmp citations comment * copy py into docker * poller grab python * wrap json parse for narrative * update delphi for create_narrative_report batch * render all cited comments * make arch ready (#2014) * make arch ready * begin splitting up cdk code * organize cdk * cdk efs and permissions fixes * docker fixes, add block storage * remove default dynamo endpoint * endpoint check fix * use us-east-1 * remove more hardcoded values * adjust delphi dockerfile * fix up setupdynamo func * fix reference * add ssl mode hardcoded * another place to add secure mode * ssl fix * update iam roles * remove hardcoded minio from docker * remove s3 check * change s3 endpoint to none if missingf * more s3 config * fix indentation * s3 logic fix * indentation fix * remove env check * comment out s3 creds * remove location constraint * unique bucket name * remove public read on aws * fix comment * endpoint urls in s3 * remove another dynamo default * more endpoint config * lint and test cleanup * more lint fix * defaults fixes * tsc fix * attempt free up disk space * multi stage builds * more space optimization * more optimization * debug * remove commented out docker code, readme and makefile updates * build fix local vs CI * pass arg to ci * add missing comma * ssl mode fixes * modify defaults and change bucket name * side by side * Mark jobs as completed. * Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py * Improve nested log handling in job poller. * add report bundles to proxy static files... * s3 for delphi routes in node server * batches running * show batch time run job timestamp * runs timestamps * quiet batch check debug logging * fix 700 s3 upload config * hash delim * show timestamp of report in ui * logging * no-op, 600 not used, but changes to topic naming * topic naming logic and display * batch uses new topic naming * clean up topic name quotes * happy memorial day bug squash * no batch limit * llm model as param * concise logging * add topic report route * topicReport shows comments * export report embeddable * export report responsiveness * deps update * viz per layer * comments report gets dropdown * switch scan command for query command * fix sorting * we have two dropdowns we should unify later * begin add GSI for perf improvements (#2029) * begin add GSI for perf improvements * refactor to avoid scan * move config to outside of handler func * replace another table scan * clean up CLI dynamo * fix 801 * cleaner batch jobs * remove another table scan * viz job client logic --------- Co-authored-by: Colin Megill <[email protected]> * actions fix * fix reports gsi query * removing zid from responses, client doesn't consume, running * update docs after checsk * viz zid * batch report rid * fix indentation issues * integrate job id * topicVizReport view * fix reportsByJobId * copy * get all layers, get consensus... * Colinmegill/node delphi (#2027) * batch debugging * batch polling * only run layer 0 * batch * Improve batch processing with new job type architecture This commit introduces a new approach to batch processing jobs with clearer job types: 1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py 2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py The change fixes a critical issue where jobs were running the wrong script, and adds proper handling of DynamoDB reserved keywords like 'status'. This state machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * narrative batch * create user fix, reset db script, commentsreport list * Unified comment extremity calculation. Stored in DynamoDB. * run delphy python conversion * switch off sh to python * call python job poller * go to create narrative batch rather than narrative batch, explicit * commentsReport linter * tmp citations comment * copy py into docker * poller grab python * wrap json parse for narrative * update delphi for create_narrative_report batch * render all cited comments * make arch ready (#2014) * make arch ready * begin splitting up cdk code * organize cdk * cdk efs and permissions fixes * docker fixes, add block storage * remove default dynamo endpoint * endpoint check fix * use us-east-1 * remove more hardcoded values * adjust delphi dockerfile * fix up setupdynamo func * fix reference * add ssl mode hardcoded * another place to add secure mode * ssl fix * update iam roles * remove hardcoded minio from docker * remove s3 check * change s3 endpoint to none if missingf * more s3 config * fix indentation * s3 logic fix * indentation fix * remove env check * comment out s3 creds * remove location constraint * unique bucket name * remove public read on aws * fix comment * endpoint urls in s3 * remove another dynamo default * more endpoint config * lint and test cleanup * more lint fix * defaults fixes * tsc fix * attempt free up disk space * multi stage builds * more space optimization * more optimization * debug * remove commented out docker code, readme and makefile updates * build fix local vs CI * pass arg to ci * add missing comma * ssl mode fixes * Mark jobs as completed. * Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py * modify defaults and change bucket name * side by side * Improve nested log handling in job poller. * add report bundles to proxy static files... * s3 for delphi routes in node server * batches running * show batch time run job timestamp * runs timestamps * quiet batch check debug logging * fix 700 s3 upload config * hash delim * show timestamp of report in ui * logging * no-op, 600 not used, but changes to topic naming * topic naming logic and display * batch uses new topic naming * clean up topic name quotes * happy memorial day bug squash * no batch limit * llm model as param * concise logging * add topic report route * topicReport shows comments * export report embeddable * export report responsiveness * remove .sh references * bump to 16 gigs default, and numba jit * remove shell script rferences from dockerfile * remove sh commands * bump python deps * run, and show, all visualizations in comments Report * claude code reference * notebooks * python tests against real data, compare with clojure * notebooks * vw analysis output * vw correlation plot * vw output * vw notebook * Revert "vw notebook" This reverts commit 7548529. * biodiversity notebook * pca validation notebook * pca validation notebook fix * add ignore, add analysis check. * gpu test * benchmark pngs * move math/python_conversion to delphi dir * setup docker for delphi * update path naming python_converstion -> delph * corrections to postgres schema * Add PostgreSQL integration tests for biodiversity dataset * Add AWS Lambda function for biodiversity dataset processing * Add optimized math functionality to delphi branch and fix PostgreSQL integration tests * Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath * Cleanup Pakistan workflow and write to DynamoDB. * Add Pakistan test for current implementation. * Add requirements. * umap narrative * move docs. * nuke test gpu direction * add evoc to gitigonre a * merge req txt * delete other req txt * remove notebooks * unified create dynamo tables * add fastapi to reqs * add missing group votes initialization * error handinling in db connect * experimental dockerization * concise readme * HAS TO BE ROOT USER!!! * run_delphi talks to dynamo internally :) * all dependencies in req.txt, and c++ in dockerfile. * run command * let's run the umap pipeline directly, wait on orchestrator * dynamo config for docker * docker endpoint for dynamo config throughout, working * ignore python init * remove enhanced table that was deprecated * add curl to dockerfile * docker compose includes ollama * run delphi script checks for ollama avail, defaults to ll3.18b * ollama generalization vs remote llm, working locally * strip quotes * claude generating report narratives. * ensure datamapplot is in scope — docker fix * run visualization after other steps, copy result out to disk * info, not debug from boto * debug model provider * generate all topics for layer 0 * generate a static plot * ignore png and svg image output from static plots * Abstract docker-compose configuration. * Integrate PCA/K-Means/Rep. pipeline. * 702 color by consesus / divisive * add 702 to run pipeline * merge dynamo and docker take 1 * endpoint dynamo * create table job * little cli * poller * Implement persistent job queue with optimistic locking - Change job table schema to use job_id as primary key - Add optimistic locking with version field - Ensure jobs are never deleted, only updated with new statuses - Implement strong consistency for critical operations - Fix DynamoDB compatibility issues with empty strings * docs for system * deprecated, now unified in top level create tables * add job status cli command * cli checks status checks status * poller logging * rename tables * docs * delphi cdk 1 * docker delphi * add narrative to dockerfile * run delphi with db fix, remove unsued file * stubbing job hierarchy * update job logs claude context * example env for ollama * log missing anthropic key * Add automatic Ollama model setup and environment variable handling for jobs * python error * ignore node modeuls for node scripts in delphi * deprecate delphi compose * parametirize ollama model for topics * complete db rename * fix list convos in cli * remove default model * get rid of default model * service url claude for client report * try catch around conversation.py * hello world move delphi to report client * Load from DynamoDB. * commentsReport route * s3 and submit job from client * no viz error message * Load data properly from DynamoDB and check results are populated. * guard llm names * docs * success showing report in commentsReport * remove cleanup script * add uuid * node batch route * Add PCAPlot component with ASCII-style symbols for high-density retro visualization * don't pop browser * batch reports endpoint * batch reports endpoint * skip static, don't pop browser * don't pop browser * batch debugging * batch polling * only run layer 0 * batch * Improve batch processing with new job type architecture This commit introduces a new approach to batch processing jobs with clearer job types: 1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py 2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py The change fixes a critical issue where jobs were running the wrong script, and adds proper handling of DynamoDB reserved keywords like 'status'. This state machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * narrative batch * create user fix, reset db script, commentsreport list * Unified comment extremity calculation. Stored in DynamoDB. * run delphy python conversion * switch off sh to python * call python job poller * go to create narrative batch rather than narrative batch, explicit * commentsReport linter * tmp citations comment * copy py into docker * poller grab python * wrap json parse for narrative * update delphi for create_narrative_report batch * render all cited comments * make arch ready (#2014) * make arch ready * begin splitting up cdk code * organize cdk * cdk efs and permissions fixes * docker fixes, add block storage * remove default dynamo endpoint * endpoint check fix * use us-east-1 * remove more hardcoded values * adjust delphi dockerfile * fix up setupdynamo func * fix reference * add ssl mode hardcoded * another place to add secure mode * ssl fix * update iam roles * remove hardcoded minio from docker * remove s3 check * change s3 endpoint to none if missingf * more s3 config * fix indentation * s3 logic fix * indentation fix * remove env check * comment out s3 creds * remove location constraint * unique bucket name * remove public read on aws * fix comment * endpoint urls in s3 * remove another dynamo default * more endpoint config * lint and test cleanup * more lint fix * defaults fixes * tsc fix * attempt free up disk space * multi stage builds * more space optimization * more optimization * debug * remove commented out docker code, readme and makefile updates * build fix local vs CI * pass arg to ci * add missing comma * ssl mode fixes * modify defaults and change bucket name * side by side * Mark jobs as completed. * Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py * Improve nested log handling in job poller. * add report bundles to proxy static files... * s3 for delphi routes in node server * batches running * show batch time run job timestamp * runs timestamps * quiet batch check debug logging * fix 700 s3 upload config * hash delim * show timestamp of report in ui * logging * no-op, 600 not used, but changes to topic naming * topic naming logic and display * batch uses new topic naming * clean up topic name quotes * happy memorial day bug squash * no batch limit * llm model as param * concise logging * add topic report route * topicReport shows comments * export report embeddable * export report responsiveness * deps update * viz per layer * comments report gets dropdown * switch scan command for query command * fix sorting * we have two dropdowns we should unify later * begin add GSI for perf improvements (#2029) * begin add GSI for perf improvements * refactor to avoid scan * move config to outside of handler func * replace another table scan * clean up CLI dynamo * fix 801 * cleaner batch jobs * remove another table scan * viz job client logic --------- Co-authored-by: Colin Megill <[email protected]> * actions fix * fix reports gsi query * removing zid from responses, client doesn't consume, running * update docs after checsk * viz zid * batch report rid * fix indentation issues * integrate job id * topicVizReport view * fix reportsByJobId * copy * attempt jest test fixes * add clear runner space to jest tests job * resolve port conflict from pg --------- Co-authored-by: Colin Megill <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Christian Weilbach <[email protected]> Co-authored-by: Bennie Rosas <[email protected]> Co-authored-by: Christian Weilbach <[email protected]> * reports rendering with new batch format * both commentsReport and topicsReport use both routes. * narrative filtering * componetize, and map jobs and sections * default seelct consensus * clean up logging * reset single convo * copywriting * match viz job id with topics job id * update cdk (#2037) * use same access key name, simplify boto3 connection (#2038) * use same access key name, simplify boto3 connection * remove duplicates * fix s3 url & py script (#2040) * bad creds fix (#2042) * bad creds fix * clean up env * remove infinite loop (#2044) * switch to on demand capacity for some tables (#2046) * switch to on demand capacity for some tables * job poll updates * poller updates final * fix invalid url bug (#2048) * claim job fix (#2050) * fix ability to claim locked jobs (#2052) * remove more bad dynamo urls (#2054) * fix date sorting logic (#2056) * poling simplification and lock cleaning (#2058) * prevent subprocess timeouts in prod (#2060) * increase timeout (#2062) * bulk test ignore * Te client side viz fixes (#2064) * find best match in key * add correct prefix * rss feed route, and topics * consensus feed rss prototype * better filtering client side (#2067) * fix key error (#2069) * upgrade delphi images (#2071) * Some delphi comments cleanup + JSON repairing (#2073) * remove unnecessary comments * json healing and comment removing * Te topic map narrative report (#2076) * begin add topicMapNarrativeReport * clean up page * Te delphi instance size config (#2077) * respect instance sizes * err fixes * make comments more responsive (#2080) * make comments more responsive * move rawdataexport to bottom * Te bulk statement upload (#2081) * begin basic bulk upload capability * save state * minor fixes * Te comment moderation 2.0 (#2084) * begin mod capability * update gemini api * fix xml output * lightening fast * clean up and fine tuning * don't analyze comment from mods * fix lint * test fix * try test fixing * add another mock * try to make mock global * try better mock path * move setup file * delete test for unused code path * try more debugging * attempt try catch * remove console log * add back deleted suite * fix instance size variable so large jobs are processed (#2087) * Te job poller size fix (#2089) * fix instance size variable so large jobs are processed * add to docker * Te job poller size fix (#2091) * fix instance size variable so large jobs are processed * add to docker * update install script * Te job poller size fix (#2093) * fix instance size variable so large jobs are processed * add to docker * update install script * replace all instances of old var * Full OIDC + JWT Integration (#2024) Full Integration of OIDC authentication system. Deprecation of legacy cookie-based auth. Upgrade client-admin app. OIDC Simulator for dev/test environments. * add missing actions vars * Fix comment filtering bug (#2111) * change from tmp (#2112) --------- Co-authored-by: Colin Megill <[email protected]> Co-authored-by: Bennie Rosas <[email protected]> Co-authored-by: Christian Weilbach <[email protected]> Co-authored-by: Christian Weilbach <[email protected]> Co-authored-by: Claude <[email protected]>
tevko
added a commit
that referenced
this pull request
Aug 6, 2025
* add ignore, add analysis check.
* gpu test
* benchmark pngs
* move math/python_conversion to delphi dir
* setup docker for delphi
* update path naming python_converstion -> delph
* corrections to postgres schema
* Add PostgreSQL integration tests for biodiversity dataset
* Add AWS Lambda function for biodiversity dataset processing
* Add optimized math functionality to delphi branch and fix PostgreSQL integration tests
* Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath
* Cleanup Pakistan workflow and write to DynamoDB.
* Add Pakistan test for current implementation.
* Add requirements.
* umap narrative
* move docs.
* nuke test gpu direction
* add evoc to gitigonre
a
* merge req txt
* delete other req txt
* remove notebooks
* unified create dynamo tables
* add fastapi to reqs
* add missing group votes initialization
* error handinling in db connect
* experimental dockerization
* concise readme
* HAS TO BE ROOT USER!!!
* run_delphi talks to dynamo internally :)
* all dependencies in req.txt, and c++ in dockerfile.
* run command
* let's run the umap pipeline directly, wait on orchestrator
* dynamo config for docker
* docker endpoint for dynamo config throughout, working
* ignore python init
* remove enhanced table that was deprecated
* add curl to dockerfile
* docker compose includes ollama
* run delphi script checks for ollama avail, defaults to ll3.18b
* ollama generalization vs remote llm, working locally
* strip quotes
* claude generating report narratives.
* ensure datamapplot is in scope — docker fix
* run visualization after other steps, copy result out to disk
* info, not debug from boto
* debug model provider
* generate all topics for layer 0
* generate a static plot
* ignore png and svg image output from static plots
* Abstract docker-compose configuration.
* Integrate PCA/K-Means/Rep. pipeline.
* 702 color by consesus / divisive
* add 702 to run pipeline
* merge dynamo and docker take 1
* endpoint dynamo
* create table job
* little cli
* poller
* Implement persistent job queue with optimistic locking
- Change job table schema to use job_id as primary key
- Add optimistic locking with version field
- Ensure jobs are never deleted, only updated with new statuses
- Implement strong consistency for critical operations
- Fix DynamoDB compatibility issues with empty strings
* docs for system
* deprecated, now unified in top level create tables
* add job status cli command
* cli checks status checks status
* poller logging
* rename tables
* docs
* delphi cdk 1
* docker delphi
* add narrative to dockerfile
* run delphi with db fix, remove unsued file
* stubbing job hierarchy
* update job logs claude context
* example env for ollama
* log missing anthropic key
* Add automatic Ollama model setup and environment variable handling for jobs
* python error
* ignore node modeuls for node scripts in delphi
* deprecate delphi compose
* parametirize ollama model for topics
* complete db rename
* fix list convos in cli
* remove default model
* get rid of default model
* service url claude for client report
* try catch around conversation.py
* hello world move delphi to report client
* Load from DynamoDB.
* commentsReport route
* s3 and submit job from client
* no viz error message
* Load data properly from DynamoDB and check results are populated.
* guard llm names
* docs
* success showing report in commentsReport
* remove cleanup script
* add uuid
* node batch route
* Add PCAPlot component with ASCII-style symbols for high-density retro visualization
* don't pop browser
* batch reports endpoint
* batch reports endpoint
* skip static, don't pop browser
* don't pop browser
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* modify defaults and change bucket name
* side by side
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* deps update
* viz per layer
* comments report gets dropdown
* switch scan command for query command
* fix sorting
* we have two dropdowns we should unify later
* begin add GSI for perf improvements (#2029)
* begin add GSI for perf improvements
* refactor to avoid scan
* move config to outside of handler func
* replace another table scan
* clean up CLI dynamo
* fix 801
* cleaner batch jobs
* remove another table scan
* viz job client logic
---------
Co-authored-by: Colin Megill <[email protected]>
* actions fix
* fix reports gsi query
* removing zid from responses, client doesn't consume, running
* update docs after checsk
* viz zid
* batch report rid
* fix indentation issues
* integrate job id
* topicVizReport view
* fix reportsByJobId
* copy
* get all layers, get consensus...
* Colinmegill/node delphi (#2027)
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* modify defaults and change bucket name
* side by side
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* remove .sh references
* bump to 16 gigs default, and numba jit
* remove shell script rferences from dockerfile
* remove sh commands
* bump python deps
* run, and show, all visualizations in comments Report
* claude code reference
* notebooks
* python tests against real data, compare with clojure
* notebooks
* vw analysis output
* vw correlation plot
* vw output
* vw notebook
* Revert "vw notebook"
This reverts commit 7548529.
* biodiversity notebook
* pca validation notebook
* pca validation notebook fix
* add ignore, add analysis check.
* gpu test
* benchmark pngs
* move math/python_conversion to delphi dir
* setup docker for delphi
* update path naming python_converstion -> delph
* corrections to postgres schema
* Add PostgreSQL integration tests for biodiversity dataset
* Add AWS Lambda function for biodiversity dataset processing
* Add optimized math functionality to delphi branch and fix PostgreSQL integration tests
* Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath
* Cleanup Pakistan workflow and write to DynamoDB.
* Add Pakistan test for current implementation.
* Add requirements.
* umap narrative
* move docs.
* nuke test gpu direction
* add evoc to gitigonre
a
* merge req txt
* delete other req txt
* remove notebooks
* unified create dynamo tables
* add fastapi to reqs
* add missing group votes initialization
* error handinling in db connect
* experimental dockerization
* concise readme
* HAS TO BE ROOT USER!!!
* run_delphi talks to dynamo internally :)
* all dependencies in req.txt, and c++ in dockerfile.
* run command
* let's run the umap pipeline directly, wait on orchestrator
* dynamo config for docker
* docker endpoint for dynamo config throughout, working
* ignore python init
* remove enhanced table that was deprecated
* add curl to dockerfile
* docker compose includes ollama
* run delphi script checks for ollama avail, defaults to ll3.18b
* ollama generalization vs remote llm, working locally
* strip quotes
* claude generating report narratives.
* ensure datamapplot is in scope — docker fix
* run visualization after other steps, copy result out to disk
* info, not debug from boto
* debug model provider
* generate all topics for layer 0
* generate a static plot
* ignore png and svg image output from static plots
* Abstract docker-compose configuration.
* Integrate PCA/K-Means/Rep. pipeline.
* 702 color by consesus / divisive
* add 702 to run pipeline
* merge dynamo and docker take 1
* endpoint dynamo
* create table job
* little cli
* poller
* Implement persistent job queue with optimistic locking
- Change job table schema to use job_id as primary key
- Add optimistic locking with version field
- Ensure jobs are never deleted, only updated with new statuses
- Implement strong consistency for critical operations
- Fix DynamoDB compatibility issues with empty strings
* docs for system
* deprecated, now unified in top level create tables
* add job status cli command
* cli checks status checks status
* poller logging
* rename tables
* docs
* delphi cdk 1
* docker delphi
* add narrative to dockerfile
* run delphi with db fix, remove unsued file
* stubbing job hierarchy
* update job logs claude context
* example env for ollama
* log missing anthropic key
* Add automatic Ollama model setup and environment variable handling for jobs
* python error
* ignore node modeuls for node scripts in delphi
* deprecate delphi compose
* parametirize ollama model for topics
* complete db rename
* fix list convos in cli
* remove default model
* get rid of default model
* service url claude for client report
* try catch around conversation.py
* hello world move delphi to report client
* Load from DynamoDB.
* commentsReport route
* s3 and submit job from client
* no viz error message
* Load data properly from DynamoDB and check results are populated.
* guard llm names
* docs
* success showing report in commentsReport
* remove cleanup script
* add uuid
* node batch route
* Add PCAPlot component with ASCII-style symbols for high-density retro visualization
* don't pop browser
* batch reports endpoint
* batch reports endpoint
* skip static, don't pop browser
* don't pop browser
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* modify defaults and change bucket name
* side by side
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* deps update
* viz per layer
* comments report gets dropdown
* switch scan command for query command
* fix sorting
* we have two dropdowns we should unify later
* begin add GSI for perf improvements (#2029)
* begin add GSI for perf improvements
* refactor to avoid scan
* move config to outside of handler func
* replace another table scan
* clean up CLI dynamo
* fix 801
* cleaner batch jobs
* remove another table scan
* viz job client logic
---------
Co-authored-by: Colin Megill <[email protected]>
* actions fix
* fix reports gsi query
* removing zid from responses, client doesn't consume, running
* update docs after checsk
* viz zid
* batch report rid
* fix indentation issues
* integrate job id
* topicVizReport view
* fix reportsByJobId
* copy
* attempt jest test fixes
* add clear runner space to jest tests job
* resolve port conflict from pg
---------
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
* reports rendering with new batch format
* both commentsReport and topicsReport use both routes.
* narrative filtering
* componetize, and map jobs and sections
* default seelct consensus
* clean up logging
* reset single convo
* copywriting
* match viz job id with topics job id
* update cdk (#2037)
* use same access key name, simplify boto3 connection (#2038)
* use same access key name, simplify boto3 connection
* remove duplicates
* fix s3 url & py script (#2040)
* bad creds fix (#2042)
* bad creds fix
* clean up env
* remove infinite loop (#2044)
* switch to on demand capacity for some tables (#2046)
* switch to on demand capacity for some tables
* job poll updates
* poller updates final
* fix invalid url bug (#2048)
* claim job fix (#2050)
* fix ability to claim locked jobs (#2052)
* remove more bad dynamo urls (#2054)
* fix date sorting logic (#2056)
* poling simplification and lock cleaning (#2058)
* prevent subprocess timeouts in prod (#2060)
* increase timeout (#2062)
* bulk test ignore
* Te client side viz fixes (#2064)
* find best match in key
* add correct prefix
* rss feed route, and topics
* consensus feed rss prototype
* better filtering client side (#2067)
* fix key error (#2069)
* upgrade delphi images (#2071)
* Some delphi comments cleanup + JSON repairing (#2073)
* remove unnecessary comments
* json healing and comment removing
* Te topic map narrative report (#2076)
* begin add topicMapNarrativeReport
* clean up page
* Te delphi instance size config (#2077)
* respect instance sizes
* err fixes
* make comments more responsive (#2080)
* make comments more responsive
* move rawdataexport to bottom
* Te bulk statement upload (#2081)
* begin basic bulk upload capability
* save state
* minor fixes
* Te comment moderation 2.0 (#2084)
* begin mod capability
* update gemini api
* fix xml output
* lightening fast
* clean up and fine tuning
* don't analyze comment from mods
* fix lint
* test fix
* try test fixing
* add another mock
* try to make mock global
* try better mock path
* move setup file
* delete test for unused code path
* try more debugging
* attempt try catch
* remove console log
* add back deleted suite
* fix instance size variable so large jobs are processed (#2087)
* Te job poller size fix (#2089)
* fix instance size variable so large jobs are processed
* add to docker
* Te job poller size fix (#2091)
* fix instance size variable so large jobs are processed
* add to docker
* update install script
* Te job poller size fix (#2093)
* fix instance size variable so large jobs are processed
* add to docker
* update install script
* replace all instances of old var
* Full OIDC + JWT Integration (#2024)
Full Integration of OIDC authentication system.
Deprecation of legacy cookie-based auth.
Upgrade client-admin app.
OIDC Simulator for dev/test environments.
* add missing actions vars
* Fix comment filtering bug (#2111)
* change from tmp (#2112)
* add reset script
* update email sending
* revert changes but with more explicit error handling
---------
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>
tevko
added a commit
that referenced
this pull request
Aug 8, 2025
* Add AWS Lambda function for biodiversity dataset processing
* Add optimized math functionality to delphi branch and fix PostgreSQL integration tests
* Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath
* Cleanup Pakistan workflow and write to DynamoDB.
* Add Pakistan test for current implementation.
* Add requirements.
* umap narrative
* move docs.
* nuke test gpu direction
* add evoc to gitigonre
a
* merge req txt
* delete other req txt
* remove notebooks
* unified create dynamo tables
* add fastapi to reqs
* add missing group votes initialization
* error handinling in db connect
* experimental dockerization
* concise readme
* HAS TO BE ROOT USER!!!
* run_delphi talks to dynamo internally :)
* all dependencies in req.txt, and c++ in dockerfile.
* run command
* let's run the umap pipeline directly, wait on orchestrator
* dynamo config for docker
* docker endpoint for dynamo config throughout, working
* ignore python init
* remove enhanced table that was deprecated
* add curl to dockerfile
* docker compose includes ollama
* run delphi script checks for ollama avail, defaults to ll3.18b
* ollama generalization vs remote llm, working locally
* strip quotes
* claude generating report narratives.
* ensure datamapplot is in scope — docker fix
* run visualization after other steps, copy result out to disk
* info, not debug from boto
* debug model provider
* generate all topics for layer 0
* generate a static plot
* ignore png and svg image output from static plots
* Abstract docker-compose configuration.
* Integrate PCA/K-Means/Rep. pipeline.
* 702 color by consesus / divisive
* add 702 to run pipeline
* merge dynamo and docker take 1
* endpoint dynamo
* create table job
* little cli
* poller
* Implement persistent job queue with optimistic locking
- Change job table schema to use job_id as primary key
- Add optimistic locking with version field
- Ensure jobs are never deleted, only updated with new statuses
- Implement strong consistency for critical operations
- Fix DynamoDB compatibility issues with empty strings
* docs for system
* deprecated, now unified in top level create tables
* add job status cli command
* cli checks status checks status
* poller logging
* rename tables
* docs
* delphi cdk 1
* docker delphi
* add narrative to dockerfile
* run delphi with db fix, remove unsued file
* stubbing job hierarchy
* update job logs claude context
* example env for ollama
* log missing anthropic key
* Add automatic Ollama model setup and environment variable handling for jobs
* python error
* ignore node modeuls for node scripts in delphi
* deprecate delphi compose
* parametirize ollama model for topics
* complete db rename
* fix list convos in cli
* remove default model
* get rid of default model
* service url claude for client report
* try catch around conversation.py
* hello world move delphi to report client
* Load from DynamoDB.
* commentsReport route
* s3 and submit job from client
* no viz error message
* Load data properly from DynamoDB and check results are populated.
* guard llm names
* docs
* success showing report in commentsReport
* remove cleanup script
* add uuid
* node batch route
* Add PCAPlot component with ASCII-style symbols for high-density retro visualization
* don't pop browser
* batch reports endpoint
* batch reports endpoint
* skip static, don't pop browser
* don't pop browser
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* modify defaults and change bucket name
* side by side
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* deps update
* viz per layer
* comments report gets dropdown
* switch scan command for query command
* fix sorting
* we have two dropdowns we should unify later
* begin add GSI for perf improvements (#2029)
* begin add GSI for perf improvements
* refactor to avoid scan
* move config to outside of handler func
* replace another table scan
* clean up CLI dynamo
* fix 801
* cleaner batch jobs
* remove another table scan
* viz job client logic
---------
Co-authored-by: Colin Megill <[email protected]>
* actions fix
* fix reports gsi query
* removing zid from responses, client doesn't consume, running
* update docs after checsk
* viz zid
* batch report rid
* fix indentation issues
* integrate job id
* topicVizReport view
* fix reportsByJobId
* copy
* get all layers, get consensus...
* Colinmegill/node delphi (#2027)
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* modify defaults and change bucket name
* side by side
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* remove .sh references
* bump to 16 gigs default, and numba jit
* remove shell script rferences from dockerfile
* remove sh commands
* bump python deps
* run, and show, all visualizations in comments Report
* claude code reference
* notebooks
* python tests against real data, compare with clojure
* notebooks
* vw analysis output
* vw correlation plot
* vw output
* vw notebook
* Revert "vw notebook"
This reverts commit 7548529.
* biodiversity notebook
* pca validation notebook
* pca validation notebook fix
* add ignore, add analysis check.
* gpu test
* benchmark pngs
* move math/python_conversion to delphi dir
* setup docker for delphi
* update path naming python_converstion -> delph
* corrections to postgres schema
* Add PostgreSQL integration tests for biodiversity dataset
* Add AWS Lambda function for biodiversity dataset processing
* Add optimized math functionality to delphi branch and fix PostgreSQL integration tests
* Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath
* Cleanup Pakistan workflow and write to DynamoDB.
* Add Pakistan test for current implementation.
* Add requirements.
* umap narrative
* move docs.
* nuke test gpu direction
* add evoc to gitigonre
a
* merge req txt
* delete other req txt
* remove notebooks
* unified create dynamo tables
* add fastapi to reqs
* add missing group votes initialization
* error handinling in db connect
* experimental dockerization
* concise readme
* HAS TO BE ROOT USER!!!
* run_delphi talks to dynamo internally :)
* all dependencies in req.txt, and c++ in dockerfile.
* run command
* let's run the umap pipeline directly, wait on orchestrator
* dynamo config for docker
* docker endpoint for dynamo config throughout, working
* ignore python init
* remove enhanced table that was deprecated
* add curl to dockerfile
* docker compose includes ollama
* run delphi script checks for ollama avail, defaults to ll3.18b
* ollama generalization vs remote llm, working locally
* strip quotes
* claude generating report narratives.
* ensure datamapplot is in scope — docker fix
* run visualization after other steps, copy result out to disk
* info, not debug from boto
* debug model provider
* generate all topics for layer 0
* generate a static plot
* ignore png and svg image output from static plots
* Abstract docker-compose configuration.
* Integrate PCA/K-Means/Rep. pipeline.
* 702 color by consesus / divisive
* add 702 to run pipeline
* merge dynamo and docker take 1
* endpoint dynamo
* create table job
* little cli
* poller
* Implement persistent job queue with optimistic locking
- Change job table schema to use job_id as primary key
- Add optimistic locking with version field
- Ensure jobs are never deleted, only updated with new statuses
- Implement strong consistency for critical operations
- Fix DynamoDB compatibility issues with empty strings
* docs for system
* deprecated, now unified in top level create tables
* add job status cli command
* cli checks status checks status
* poller logging
* rename tables
* docs
* delphi cdk 1
* docker delphi
* add narrative to dockerfile
* run delphi with db fix, remove unsued file
* stubbing job hierarchy
* update job logs claude context
* example env for ollama
* log missing anthropic key
* Add automatic Ollama model setup and environment variable handling for jobs
* python error
* ignore node modeuls for node scripts in delphi
* deprecate delphi compose
* parametirize ollama model for topics
* complete db rename
* fix list convos in cli
* remove default model
* get rid of default model
* service url claude for client report
* try catch around conversation.py
* hello world move delphi to report client
* Load from DynamoDB.
* commentsReport route
* s3 and submit job from client
* no viz error message
* Load data properly from DynamoDB and check results are populated.
* guard llm names
* docs
* success showing report in commentsReport
* remove cleanup script
* add uuid
* node batch route
* Add PCAPlot component with ASCII-style symbols for high-density retro visualization
* don't pop browser
* batch reports endpoint
* batch reports endpoint
* skip static, don't pop browser
* don't pop browser
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* modify defaults and change bucket name
* side by side
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* deps update
* viz per layer
* comments report gets dropdown
* switch scan command for query command
* fix sorting
* we have two dropdowns we should unify later
* begin add GSI for perf improvements (#2029)
* begin add GSI for perf improvements
* refactor to avoid scan
* move config to outside of handler func
* replace another table scan
* clean up CLI dynamo
* fix 801
* cleaner batch jobs
* remove another table scan
* viz job client logic
---------
Co-authored-by: Colin Megill <[email protected]>
* actions fix
* fix reports gsi query
* removing zid from responses, client doesn't consume, running
* update docs after checsk
* viz zid
* batch report rid
* fix indentation issues
* integrate job id
* topicVizReport view
* fix reportsByJobId
* copy
* attempt jest test fixes
* add clear runner space to jest tests job
* resolve port conflict from pg
---------
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
* reports rendering with new batch format
* both commentsReport and topicsReport use both routes.
* narrative filtering
* componetize, and map jobs and sections
* default seelct consensus
* clean up logging
* reset single convo
* copywriting
* match viz job id with topics job id
* update cdk (#2037)
* use same access key name, simplify boto3 connection (#2038)
* use same access key name, simplify boto3 connection
* remove duplicates
* fix s3 url & py script (#2040)
* bad creds fix (#2042)
* bad creds fix
* clean up env
* remove infinite loop (#2044)
* switch to on demand capacity for some tables (#2046)
* switch to on demand capacity for some tables
* job poll updates
* poller updates final
* fix invalid url bug (#2048)
* claim job fix (#2050)
* fix ability to claim locked jobs (#2052)
* remove more bad dynamo urls (#2054)
* fix date sorting logic (#2056)
* poling simplification and lock cleaning (#2058)
* prevent subprocess timeouts in prod (#2060)
* increase timeout (#2062)
* bulk test ignore
* Te client side viz fixes (#2064)
* find best match in key
* add correct prefix
* rss feed route, and topics
* consensus feed rss prototype
* better filtering client side (#2067)
* fix key error (#2069)
* upgrade delphi images (#2071)
* Some delphi comments cleanup + JSON repairing (#2073)
* remove unnecessary comments
* json healing and comment removing
* Te topic map narrative report (#2076)
* begin add topicMapNarrativeReport
* clean up page
* Te delphi instance size config (#2077)
* respect instance sizes
* err fixes
* make comments more responsive (#2080)
* make comments more responsive
* move rawdataexport to bottom
* Te bulk statement upload (#2081)
* begin basic bulk upload capability
* save state
* minor fixes
* Te comment moderation 2.0 (#2084)
* begin mod capability
* update gemini api
* fix xml output
* lightening fast
* clean up and fine tuning
* don't analyze comment from mods
* fix lint
* test fix
* try test fixing
* add another mock
* try to make mock global
* try better mock path
* move setup file
* delete test for unused code path
* try more debugging
* attempt try catch
* remove console log
* add back deleted suite
* fix instance size variable so large jobs are processed (#2087)
* Te job poller size fix (#2089)
* fix instance size variable so large jobs are processed
* add to docker
* Te job poller size fix (#2091)
* fix instance size variable so large jobs are processed
* add to docker
* update install script
* Te job poller size fix (#2093)
* fix instance size variable so large jobs are processed
* add to docker
* update install script
* replace all instances of old var
* Full OIDC + JWT Integration (#2024)
Full Integration of OIDC authentication system.
Deprecation of legacy cookie-based auth.
Upgrade client-admin app.
OIDC Simulator for dev/test environments.
* add missing actions vars
* Fix comment filtering bug (#2111)
* change from tmp (#2112)
* add reset script
* update email sending
* revert changes but with more explicit error handling
* Te client participation 2 (#2098)
* add astro version of client-participation
* minor style updates
* begin moving over translations
* port over translations
* ssr
* Use real convo data
* dynamically update title
* more translations
* wire in voting capability
* get voting working
* cors
* type check
* impliment convo closed status
* admin mod, prio stub
* hclust route checkpoitn
* labels working
* remove logging, get towards hclust
* hclust explration 22
* contour plot
* contours
* prio take 1
* distances
* remove dead 600 file
* add more defense for small models
* topic prioritize mock
* webpack build size
* topic sections
* merge fix
* add simple route
* Restructure priority calculation to use group-based extremity
- Remove priority calculation from math pipeline (conversation.py)
* Delete _importance_metric and _priority_metric methods
* Remove priority computation from recompute() method
* Math pipeline now focuses on PCA, clustering, and representativeness
- Add dedicated priority calculation script (502_calculate_priorities.py)
* Implements PriorityCalculator class with group-based extremity
* Matches Clojure priority formula: (importance * scaling_factor)^2
* Retrieves extremity values from Delphi_CommentExtremity table
* Updates priorities in Delphi_CommentRouting table
- Update pipeline execution order (run_delphi.py, run_delphi.sh)
* Math pipeline → UMAP pipeline → Extremity calculation → Priority calculation
* Ensures priorities use group-based extremity instead of PCA-based
* Maintains separation of concerns between mathematical and priority calculations
This fixes the priority calculation bug where all priorities were 0 due to
missing extremity values, and implements proper group-based extremity usage
as requested for the Pakistan conversation analysis.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* Refactor priority calculation: separate formulas from service logic
- Extract priority formulas into polismath/conversation/priority.py
* Create PriorityCalculator class with static methods for core formulas
* Port importance_metric and priority_metric from Clojure implementation
* Add convenience methods: calculate_comment_priority, validate_inputs, explain_priority
* Pure mathematical logic with no I/O dependencies for better testability
- Refactor umap_narrative/502_calculate_priorities.py to use extracted formulas
* Rename class from PriorityCalculator to PriorityService (clearer distinction)
* Import and use PriorityCalculator.calculate_comment_priority()
* Remove duplicate formula implementations (38 lines removed)
* Focus service on DynamoDB operations and data orchestration
Benefits:
- Better separation of concerns: formulas vs data processing
- Improved testability: mathematical logic can be unit tested independently
- Enhanced reusability: priority formulas can be used in other contexts
- Cleaner maintainability: formula changes only need to happen in one place
Tested successfully with conversation 36324:
- 807 comments processed
- Priority statistics: min=0, max=3, avg=2.36
- All priorities calculated using group-based extremity values
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* Revert "Refactor priority calculation: separate formulas from service logic"
This reverts commit cfd829d.
* topic agenda collapse prototype tetris-y
* topic agenda setting demo
* begin basic bulk upload capability
* save state
* split by distance doesn't owrk
* sentence transformer model
* componetize
* drive detail
* more specifics
* log archetypal comments
* dynamo table and node services
* Don't crash when we have not votes.
* suppress logging
* differnet approach for prepended labesl
* Te arch updates topicmod (#2086)
* add polis mod access and remove table scans
* add comments back
* more comment fixes
* add footer links
* begin dockerization
* dockerize and new route structure
* wire in xid
* get alpha embed working
* prune comments
* connect comment form
* translations fixes, hook up subscribe form
* css fixes
* fix env vars
* Colinmegill/topic prio (#2103)
* showing topics in participant view
* logo color / p slightly more minimal change
---------
Co-authored-by: Colin Megill <[email protected]>
* perf adjustments
* remove console log
* minor fixes
* compilation fixes
* more auth
* get auth working
* return vote response
* participation alpha qa and better topic loading
* add hide header ability
* update headerless
* turn off eslint for now
* prep alpha for prod deploy
---------
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>
* update compose-test
* fix test compose network name
* compose test cert volume
* update test action
* add cpa to test file
* gh actions copy certs for tests
---------
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>
tevko
added a commit
that referenced
this pull request
Aug 8, 2025
* Cleanup Pakistan workflow and write to DynamoDB.
* Add Pakistan test for current implementation.
* Add requirements.
* umap narrative
* move docs.
* nuke test gpu direction
* add evoc to gitigonre
a
* merge req txt
* delete other req txt
* remove notebooks
* unified create dynamo tables
* add fastapi to reqs
* add missing group votes initialization
* error handinling in db connect
* experimental dockerization
* concise readme
* HAS TO BE ROOT USER!!!
* run_delphi talks to dynamo internally :)
* all dependencies in req.txt, and c++ in dockerfile.
* run command
* let's run the umap pipeline directly, wait on orchestrator
* dynamo config for docker
* docker endpoint for dynamo config throughout, working
* ignore python init
* remove enhanced table that was deprecated
* add curl to dockerfile
* docker compose includes ollama
* run delphi script checks for ollama avail, defaults to ll3.18b
* ollama generalization vs remote llm, working locally
* strip quotes
* claude generating report narratives.
* ensure datamapplot is in scope — docker fix
* run visualization after other steps, copy result out to disk
* info, not debug from boto
* debug model provider
* generate all topics for layer 0
* generate a static plot
* ignore png and svg image output from static plots
* Abstract docker-compose configuration.
* Integrate PCA/K-Means/Rep. pipeline.
* 702 color by consesus / divisive
* add 702 to run pipeline
* merge dynamo and docker take 1
* endpoint dynamo
* create table job
* little cli
* poller
* Implement persistent job queue with optimistic locking
- Change job table schema to use job_id as primary key
- Add optimistic locking with version field
- Ensure jobs are never deleted, only updated with new statuses
- Implement strong consistency for critical operations
- Fix DynamoDB compatibility issues with empty strings
* docs for system
* deprecated, now unified in top level create tables
* add job status cli command
* cli checks status checks status
* poller logging
* rename tables
* docs
* delphi cdk 1
* docker delphi
* add narrative to dockerfile
* run delphi with db fix, remove unsued file
* stubbing job hierarchy
* update job logs claude context
* example env for ollama
* log missing anthropic key
* Add automatic Ollama model setup and environment variable handling for jobs
* python error
* ignore node modeuls for node scripts in delphi
* deprecate delphi compose
* parametirize ollama model for topics
* complete db rename
* fix list convos in cli
* remove default model
* get rid of default model
* service url claude for client report
* try catch around conversation.py
* hello world move delphi to report client
* Load from DynamoDB.
* commentsReport route
* s3 and submit job from client
* no viz error message
* Load data properly from DynamoDB and check results are populated.
* guard llm names
* docs
* success showing report in commentsReport
* remove cleanup script
* add uuid
* node batch route
* Add PCAPlot component with ASCII-style symbols for high-density retro visualization
* don't pop browser
* batch reports endpoint
* batch reports endpoint
* skip static, don't pop browser
* don't pop browser
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* modify defaults and change bucket name
* side by side
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* deps update
* viz per layer
* comments report gets dropdown
* switch scan command for query command
* fix sorting
* we have two dropdowns we should unify later
* begin add GSI for perf improvements (#2029)
* begin add GSI for perf improvements
* refactor to avoid scan
* move config to outside of handler func
* replace another table scan
* clean up CLI dynamo
* fix 801
* cleaner batch jobs
* remove another table scan
* viz job client logic
---------
Co-authored-by: Colin Megill <[email protected]>
* actions fix
* fix reports gsi query
* removing zid from responses, client doesn't consume, running
* update docs after checsk
* viz zid
* batch report rid
* fix indentation issues
* integrate job id
* topicVizReport view
* fix reportsByJobId
* copy
* get all layers, get consensus...
* Colinmegill/node delphi (#2027)
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* modify defaults and change bucket name
* side by side
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* remove .sh references
* bump to 16 gigs default, and numba jit
* remove shell script rferences from dockerfile
* remove sh commands
* bump python deps
* run, and show, all visualizations in comments Report
* claude code reference
* notebooks
* python tests against real data, compare with clojure
* notebooks
* vw analysis output
* vw correlation plot
* vw output
* vw notebook
* Revert "vw notebook"
This reverts commit 7548529.
* biodiversity notebook
* pca validation notebook
* pca validation notebook fix
* add ignore, add analysis check.
* gpu test
* benchmark pngs
* move math/python_conversion to delphi dir
* setup docker for delphi
* update path naming python_converstion -> delph
* corrections to postgres schema
* Add PostgreSQL integration tests for biodiversity dataset
* Add AWS Lambda function for biodiversity dataset processing
* Add optimized math functionality to delphi branch and fix PostgreSQL integration tests
* Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath
* Cleanup Pakistan workflow and write to DynamoDB.
* Add Pakistan test for current implementation.
* Add requirements.
* umap narrative
* move docs.
* nuke test gpu direction
* add evoc to gitigonre
a
* merge req txt
* delete other req txt
* remove notebooks
* unified create dynamo tables
* add fastapi to reqs
* add missing group votes initialization
* error handinling in db connect
* experimental dockerization
* concise readme
* HAS TO BE ROOT USER!!!
* run_delphi talks to dynamo internally :)
* all dependencies in req.txt, and c++ in dockerfile.
* run command
* let's run the umap pipeline directly, wait on orchestrator
* dynamo config for docker
* docker endpoint for dynamo config throughout, working
* ignore python init
* remove enhanced table that was deprecated
* add curl to dockerfile
* docker compose includes ollama
* run delphi script checks for ollama avail, defaults to ll3.18b
* ollama generalization vs remote llm, working locally
* strip quotes
* claude generating report narratives.
* ensure datamapplot is in scope — docker fix
* run visualization after other steps, copy result out to disk
* info, not debug from boto
* debug model provider
* generate all topics for layer 0
* generate a static plot
* ignore png and svg image output from static plots
* Abstract docker-compose configuration.
* Integrate PCA/K-Means/Rep. pipeline.
* 702 color by consesus / divisive
* add 702 to run pipeline
* merge dynamo and docker take 1
* endpoint dynamo
* create table job
* little cli
* poller
* Implement persistent job queue with optimistic locking
- Change job table schema to use job_id as primary key
- Add optimistic locking with version field
- Ensure jobs are never deleted, only updated with new statuses
- Implement strong consistency for critical operations
- Fix DynamoDB compatibility issues with empty strings
* docs for system
* deprecated, now unified in top level create tables
* add job status cli command
* cli checks status checks status
* poller logging
* rename tables
* docs
* delphi cdk 1
* docker delphi
* add narrative to dockerfile
* run delphi with db fix, remove unsued file
* stubbing job hierarchy
* update job logs claude context
* example env for ollama
* log missing anthropic key
* Add automatic Ollama model setup and environment variable handling for jobs
* python error
* ignore node modeuls for node scripts in delphi
* deprecate delphi compose
* parametirize ollama model for topics
* complete db rename
* fix list convos in cli
* remove default model
* get rid of default model
* service url claude for client report
* try catch around conversation.py
* hello world move delphi to report client
* Load from DynamoDB.
* commentsReport route
* s3 and submit job from client
* no viz error message
* Load data properly from DynamoDB and check results are populated.
* guard llm names
* docs
* success showing report in commentsReport
* remove cleanup script
* add uuid
* node batch route
* Add PCAPlot component with ASCII-style symbols for high-density retro visualization
* don't pop browser
* batch reports endpoint
* batch reports endpoint
* skip static, don't pop browser
* don't pop browser
* batch debugging
* batch polling
* only run layer 0
* batch
* Improve batch processing with new job type architecture
This commit introduces a new approach to batch processing jobs with clearer job types:
1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py
The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* narrative batch
* create user fix, reset db script, commentsreport list
* Unified comment extremity calculation. Stored in DynamoDB.
* run delphy python conversion
* switch off sh to python
* call python job poller
* go to create narrative batch rather than narrative batch, explicit
* commentsReport linter
* tmp citations comment
* copy py into docker
* poller grab python
* wrap json parse for narrative
* update delphi for create_narrative_report batch
* render all cited comments
* make arch ready (#2014)
* make arch ready
* begin splitting up cdk code
* organize cdk
* cdk efs and permissions fixes
* docker fixes, add block storage
* remove default dynamo endpoint
* endpoint check fix
* use us-east-1
* remove more hardcoded values
* adjust delphi dockerfile
* fix up setupdynamo func
* fix reference
* add ssl mode hardcoded
* another place to add secure mode
* ssl fix
* update iam roles
* remove hardcoded minio from docker
* remove s3 check
* change s3 endpoint to none if missingf
* more s3 config
* fix indentation
* s3 logic fix
* indentation fix
* remove env check
* comment out s3 creds
* remove location constraint
* unique bucket name
* remove public read on aws
* fix comment
* endpoint urls in s3
* remove another dynamo default
* more endpoint config
* lint and test cleanup
* more lint fix
* defaults fixes
* tsc fix
* attempt free up disk space
* multi stage builds
* more space optimization
* more optimization
* debug
* remove commented out docker code, readme and makefile updates
* build fix local vs CI
* pass arg to ci
* add missing comma
* ssl mode fixes
* modify defaults and change bucket name
* side by side
* Mark jobs as completed.
* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py
* Improve nested log handling in job poller.
* add report bundles to proxy static files...
* s3 for delphi routes in node server
* batches running
* show batch time run job timestamp
* runs timestamps
* quiet batch check debug logging
* fix 700 s3 upload config
* hash delim
* show timestamp of report in ui
* logging
* no-op, 600 not used, but changes to topic naming
* topic naming logic and display
* batch uses new topic naming
* clean up topic name quotes
* happy memorial day bug squash
* no batch limit
* llm model as param
* concise logging
* add topic report route
* topicReport shows comments
* export report embeddable
* export report responsiveness
* deps update
* viz per layer
* comments report gets dropdown
* switch scan command for query command
* fix sorting
* we have two dropdowns we should unify later
* begin add GSI for perf improvements (#2029)
* begin add GSI for perf improvements
* refactor to avoid scan
* move config to outside of handler func
* replace another table scan
* clean up CLI dynamo
* fix 801
* cleaner batch jobs
* remove another table scan
* viz job client logic
---------
Co-authored-by: Colin Megill <[email protected]>
* actions fix
* fix reports gsi query
* removing zid from responses, client doesn't consume, running
* update docs after checsk
* viz zid
* batch report rid
* fix indentation issues
* integrate job id
* topicVizReport view
* fix reportsByJobId
* copy
* attempt jest test fixes
* add clear runner space to jest tests job
* resolve port conflict from pg
---------
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
* reports rendering with new batch format
* both commentsReport and topicsReport use both routes.
* narrative filtering
* componetize, and map jobs and sections
* default seelct consensus
* clean up logging
* reset single convo
* copywriting
* match viz job id with topics job id
* update cdk (#2037)
* use same access key name, simplify boto3 connection (#2038)
* use same access key name, simplify boto3 connection
* remove duplicates
* fix s3 url & py script (#2040)
* bad creds fix (#2042)
* bad creds fix
* clean up env
* remove infinite loop (#2044)
* switch to on demand capacity for some tables (#2046)
* switch to on demand capacity for some tables
* job poll updates
* poller updates final
* fix invalid url bug (#2048)
* claim job fix (#2050)
* fix ability to claim locked jobs (#2052)
* remove more bad dynamo urls (#2054)
* fix date sorting logic (#2056)
* poling simplification and lock cleaning (#2058)
* prevent subprocess timeouts in prod (#2060)
* increase timeout (#2062)
* bulk test ignore
* Te client side viz fixes (#2064)
* find best match in key
* add correct prefix
* rss feed route, and topics
* consensus feed rss prototype
* better filtering client side (#2067)
* fix key error (#2069)
* upgrade delphi images (#2071)
* Some delphi comments cleanup + JSON repairing (#2073)
* remove unnecessary comments
* json healing and comment removing
* Te topic map narrative report (#2076)
* begin add topicMapNarrativeReport
* clean up page
* Te delphi instance size config (#2077)
* respect instance sizes
* err fixes
* make comments more responsive (#2080)
* make comments more responsive
* move rawdataexport to bottom
* Te bulk statement upload (#2081)
* begin basic bulk upload capability
* save state
* minor fixes
* Te comment moderation 2.0 (#2084)
* begin mod capability
* update gemini api
* fix xml output
* lightening fast
* clean up and fine tuning
* don't analyze comment from mods
* fix lint
* test fix
* try test fixing
* add another mock
* try to make mock global
* try better mock path
* move setup file
* delete test for unused code path
* try more debugging
* attempt try catch
* remove console log
* add back deleted suite
* fix instance size variable so large jobs are processed (#2087)
* Te job poller size fix (#2089)
* fix instance size variable so large jobs are processed
* add to docker
* Te job poller size fix (#2091)
* fix instance size variable so large jobs are processed
* add to docker
* update install script
* Te job poller size fix (#2093)
* fix instance size variable so large jobs are processed
* add to docker
* update install script
* replace all instances of old var
* Full OIDC + JWT Integration (#2024)
Full Integration of OIDC authentication system.
Deprecation of legacy cookie-based auth.
Upgrade client-admin app.
OIDC Simulator for dev/test environments.
* add missing actions vars
* Fix comment filtering bug (#2111)
* change from tmp (#2112)
* add reset script
* update email sending
* revert changes but with more explicit error handling
* Te client participation 2 (#2098)
* add astro version of client-participation
* minor style updates
* begin moving over translations
* port over translations
* ssr
* Use real convo data
* dynamically update title
* more translations
* wire in voting capability
* get voting working
* cors
* type check
* impliment convo closed status
* admin mod, prio stub
* hclust route checkpoitn
* labels working
* remove logging, get towards hclust
* hclust explration 22
* contour plot
* contours
* prio take 1
* distances
* remove dead 600 file
* add more defense for small models
* topic prioritize mock
* webpack build size
* topic sections
* merge fix
* add simple route
* Restructure priority calculation to use group-based extremity
- Remove priority calculation from math pipeline (conversation.py)
* Delete _importance_metric and _priority_metric methods
* Remove priority computation from recompute() method
* Math pipeline now focuses on PCA, clustering, and representativeness
- Add dedicated priority calculation script (502_calculate_priorities.py)
* Implements PriorityCalculator class with group-based extremity
* Matches Clojure priority formula: (importance * scaling_factor)^2
* Retrieves extremity values from Delphi_CommentExtremity table
* Updates priorities in Delphi_CommentRouting table
- Update pipeline execution order (run_delphi.py, run_delphi.sh)
* Math pipeline → UMAP pipeline → Extremity calculation → Priority calculation
* Ensures priorities use group-based extremity instead of PCA-based
* Maintains separation of concerns between mathematical and priority calculations
This fixes the priority calculation bug where all priorities were 0 due to
missing extremity values, and implements proper group-based extremity usage
as requested for the Pakistan conversation analysis.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* Refactor priority calculation: separate formulas from service logic
- Extract priority formulas into polismath/conversation/priority.py
* Create PriorityCalculator class with static methods for core formulas
* Port importance_metric and priority_metric from Clojure implementation
* Add convenience methods: calculate_comment_priority, validate_inputs, explain_priority
* Pure mathematical logic with no I/O dependencies for better testability
- Refactor umap_narrative/502_calculate_priorities.py to use extracted formulas
* Rename class from PriorityCalculator to PriorityService (clearer distinction)
* Import and use PriorityCalculator.calculate_comment_priority()
* Remove duplicate formula implementations (38 lines removed)
* Focus service on DynamoDB operations and data orchestration
Benefits:
- Better separation of concerns: formulas vs data processing
- Improved testability: mathematical logic can be unit tested independently
- Enhanced reusability: priority formulas can be used in other contexts
- Cleaner maintainability: formula changes only need to happen in one place
Tested successfully with conversation 36324:
- 807 comments processed
- Priority statistics: min=0, max=3, avg=2.36
- All priorities calculated using group-based extremity values
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <[email protected]>
* Revert "Refactor priority calculation: separate formulas from service logic"
This reverts commit cfd829d.
* topic agenda collapse prototype tetris-y
* topic agenda setting demo
* begin basic bulk upload capability
* save state
* split by distance doesn't owrk
* sentence transformer model
* componetize
* drive detail
* more specifics
* log archetypal comments
* dynamo table and node services
* Don't crash when we have not votes.
* suppress logging
* differnet approach for prepended labesl
* Te arch updates topicmod (#2086)
* add polis mod access and remove table scans
* add comments back
* more comment fixes
* add footer links
* begin dockerization
* dockerize and new route structure
* wire in xid
* get alpha embed working
* prune comments
* connect comment form
* translations fixes, hook up subscribe form
* css fixes
* fix env vars
* Colinmegill/topic prio (#2103)
* showing topics in participant view
* logo color / p slightly more minimal change
---------
Co-authored-by: Colin Megill <[email protected]>
* perf adjustments
* remove console log
* minor fixes
* compilation fixes
* more auth
* get auth working
* return vote response
* participation alpha qa and better topic loading
* add hide header ability
* update headerless
* turn off eslint for now
* prep alpha for prod deploy
---------
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>
* update compose-test
* fix test compose network name
* compose test cert volume
* update test action
* add cpa to test file
* gh actions copy certs for tests
* fix token in header
* update token logic
---------
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Root cause
There was redundancy in comment filtering, using either
not_voted_by=pidand justpid=pidto filter out comments voted by a participant. The old code was probably not including pid in the request object normally. Now with the new auth and better participant handling, pid is generally present in the request and should not be used to filter comments.Solution
Remove filtering by pid UNLESS
not_voted_by=pidis explicitly included. This is the expected behavior.Additional changes
CommentTypetoGetCommentsParams