* Add AWS Lambda function for biodiversity dataset processing

* Add optimized math functionality to delphi branch and fix PostgreSQL integration tests

* Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath

* Cleanup Pakistan workflow and write to DynamoDB.

* Add Pakistan test for current implementation.

* Add requirements.

* umap narrative

* move docs.

* nuke test gpu direction

* add evoc to gitigonre

a

* merge req txt

* delete other req txt

* remove notebooks

* unified create dynamo tables

* add fastapi to reqs

* add missing group votes initialization

* error handinling in db connect

* experimental dockerization

* concise readme

* HAS TO BE ROOT USER!!!

* run_delphi talks to dynamo internally :)

* all dependencies in req.txt, and c++ in dockerfile.

* run command

* let's run the umap pipeline directly, wait on orchestrator

* dynamo config for docker

* docker endpoint for dynamo config throughout, working

* ignore python init

* remove enhanced table that was deprecated

* add curl to dockerfile

* docker compose includes ollama

* run delphi script checks for ollama avail, defaults to ll3.18b

* ollama generalization vs remote llm, working locally

* strip quotes

* claude generating report narratives.

* ensure datamapplot is in scope — docker fix

* run visualization after other steps, copy result out to disk

* info, not debug from boto

* debug model provider

* generate all topics for layer 0

* generate a static plot

* ignore png and svg image output from static plots

* Abstract docker-compose configuration.

* Integrate PCA/K-Means/Rep. pipeline.

* 702 color by consesus / divisive

* add 702 to run pipeline

* merge dynamo and docker take 1

* endpoint dynamo

* create table job

* little cli

* poller

* Implement persistent job queue with optimistic locking

- Change job table schema to use job_id as primary key
      - Add optimistic locking with version field
      - Ensure jobs are never deleted, only updated with new statuses
      - Implement strong consistency for critical operations
      - Fix DynamoDB compatibility issues with empty strings

* docs for system

* deprecated, now unified in top level create tables

* add job status cli command

* cli checks status checks status

* poller logging

* rename tables

* docs

* delphi cdk 1

* docker delphi

* add narrative to dockerfile

* run delphi with db fix, remove unsued file

* stubbing job hierarchy

* update job logs claude context

* example env for ollama

* log missing anthropic key

* Add automatic Ollama model setup and environment variable handling for jobs

* python error

* ignore node modeuls for node scripts in delphi

* deprecate delphi compose

* parametirize ollama model for topics

* complete db rename

* fix list convos in cli

* remove default model

* get rid of default model

* service url claude for client report

* try catch around conversation.py

* hello world move delphi to report client

* Load from DynamoDB.

* commentsReport route

* s3 and submit job from client

* no viz error message

* Load data properly from DynamoDB and check results are populated.

* guard llm names

* docs

* success showing report in commentsReport

* remove cleanup script

* add uuid

* node batch route

* Add PCAPlot component with ASCII-style symbols for high-density retro visualization

* don't pop browser

* batch reports endpoint

* batch reports endpoint

* skip static, don't pop browser

* don't pop browser

* batch debugging

* batch polling

* only run layer 0

* batch

* Improve batch processing with new job type architecture

This commit introduces a new approach to batch processing jobs with clearer job types:

1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py

The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* narrative batch

* create user fix, reset db script, commentsreport list

* Unified comment extremity calculation. Stored in DynamoDB.

* run delphy python conversion

* switch off sh to python

* call python job poller

* go to create narrative batch rather than narrative batch, explicit

* commentsReport linter

* tmp citations comment

* copy py into docker

* poller grab python

* wrap json parse for narrative

* update delphi for create_narrative_report batch

* render all cited comments

* make arch ready (#2014)

* make arch ready

* begin splitting up cdk code

* organize cdk

* cdk efs and permissions fixes

* docker fixes, add block storage

* remove default dynamo endpoint

* endpoint check fix

* use us-east-1

* remove more hardcoded values

* adjust delphi dockerfile

* fix up setupdynamo func

* fix reference

* add ssl mode hardcoded

* another place to add secure mode

* ssl fix

* update iam roles

* remove hardcoded minio from docker

* remove s3 check

* change s3 endpoint to none if missingf

* more s3 config

* fix indentation

* s3 logic fix

* indentation fix

* remove env check

* comment out s3 creds

* remove location constraint

* unique bucket name

* remove public read on aws

* fix comment

* endpoint urls in s3

* remove another dynamo default

* more endpoint config

* lint and test cleanup

* more lint fix

* defaults fixes

* tsc fix

* attempt free up disk space

* multi stage builds

* more space optimization

* more optimization

* debug

* remove commented out docker code, readme and makefile updates

* build fix local vs CI

* pass arg to ci

* add missing comma

* ssl mode fixes

* modify defaults and change bucket name

* side by side

* Mark jobs as completed.

* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py

* Improve nested log handling in job poller.

* add report bundles to proxy static files...

* s3 for delphi routes in node server

* batches running

* show batch time run job timestamp

* runs timestamps

* quiet batch check debug logging

* fix 700 s3 upload config

* hash delim

* show timestamp of report in ui

* logging

* no-op, 600 not used, but changes to topic naming

* topic naming logic and display

* batch uses new topic naming

* clean up topic name quotes

* happy memorial day bug squash

* no batch limit

* llm model as param

* concise logging

* add topic report route

* topicReport shows comments

* export report embeddable

* export report responsiveness

* deps update

* viz per layer

* comments report gets dropdown

* switch scan command for query command

* fix sorting

* we have two dropdowns we should unify later

* begin add GSI for perf improvements (#2029)

* begin add GSI for perf improvements

* refactor to avoid scan

* move config to outside of handler func

* replace another table scan

* clean up CLI dynamo

* fix 801

* cleaner batch jobs

* remove another table scan

* viz job client logic

---------

Co-authored-by: Colin Megill <[email protected]>

* actions fix

* fix reports gsi query

* removing zid from responses, client doesn't consume, running

* update docs after checsk

* viz zid

* batch report rid

* fix indentation issues

* integrate job id

* topicVizReport view

* fix reportsByJobId

* copy

* get all layers, get consensus...

* Colinmegill/node delphi (#2027)

* batch debugging

* batch polling

* only run layer 0

* batch

* Improve batch processing with new job type architecture

This commit introduces a new approach to batch processing jobs with clearer job types:

1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py

The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* narrative batch

* create user fix, reset db script, commentsreport list

* Unified comment extremity calculation. Stored in DynamoDB.

* run delphy python conversion

* switch off sh to python

* call python job poller

* go to create narrative batch rather than narrative batch, explicit

* commentsReport linter

* tmp citations comment

* copy py into docker

* poller grab python

* wrap json parse for narrative

* update delphi for create_narrative_report batch

* render all cited comments

* make arch ready (#2014)

* make arch ready

* begin splitting up cdk code

* organize cdk

* cdk efs and permissions fixes

* docker fixes, add block storage

* remove default dynamo endpoint

* endpoint check fix

* use us-east-1

* remove more hardcoded values

* adjust delphi dockerfile

* fix up setupdynamo func

* fix reference

* add ssl mode hardcoded

* another place to add secure mode

* ssl fix

* update iam roles

* remove hardcoded minio from docker

* remove s3 check

* change s3 endpoint to none if missingf

* more s3 config

* fix indentation

* s3 logic fix

* indentation fix

* remove env check

* comment out s3 creds

* remove location constraint

* unique bucket name

* remove public read on aws

* fix comment

* endpoint urls in s3

* remove another dynamo default

* more endpoint config

* lint and test cleanup

* more lint fix

* defaults fixes

* tsc fix

* attempt free up disk space

* multi stage builds

* more space optimization

* more optimization

* debug

* remove commented out docker code, readme and makefile updates

* build fix local vs CI

* pass arg to ci

* add missing comma

* ssl mode fixes

* Mark jobs as completed.

* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py

* modify defaults and change bucket name

* side by side

* Improve nested log handling in job poller.

* add report bundles to proxy static files...

* s3 for delphi routes in node server

* batches running

* show batch time run job timestamp

* runs timestamps

* quiet batch check debug logging

* fix 700 s3 upload config

* hash delim

* show timestamp of report in ui

* logging

* no-op, 600 not used, but changes to topic naming

* topic naming logic and display

* batch uses new topic naming

* clean up topic name quotes

* happy memorial day bug squash

* no batch limit

* llm model as param

* concise logging

* add topic report route

* topicReport shows comments

* export report embeddable

* export report responsiveness

* remove .sh references

* bump to 16 gigs default, and numba jit

* remove shell script rferences from dockerfile

* remove sh commands

* bump python deps

* run, and show, all visualizations in comments Report

* claude code reference

* notebooks

* python tests against real data, compare with clojure

* notebooks

* vw analysis output

* vw correlation plot

* vw output

* vw notebook

* Revert "vw notebook"

This reverts commit 7548529.

* biodiversity notebook

* pca validation notebook

* pca validation notebook fix

* add ignore, add analysis check.

* gpu test

* benchmark pngs

* move math/python_conversion to delphi dir

* setup docker for delphi

* update path naming python_converstion -> delph

* corrections to postgres schema

* Add PostgreSQL integration tests for biodiversity dataset

* Add AWS Lambda function for biodiversity dataset processing

* Add optimized math functionality to delphi branch and fix PostgreSQL integration tests

* Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath

* Cleanup Pakistan workflow and write to DynamoDB.

* Add Pakistan test for current implementation.

* Add requirements.

* umap narrative

* move docs.

* nuke test gpu direction

* add evoc to gitigonre

a

* merge req txt

* delete other req txt

* remove notebooks

* unified create dynamo tables

* add fastapi to reqs

* add missing group votes initialization

* error handinling in db connect

* experimental dockerization

* concise readme

* HAS TO BE ROOT USER!!!

* run_delphi talks to dynamo internally :)

* all dependencies in req.txt, and c++ in dockerfile.

* run command

* let's run the umap pipeline directly, wait on orchestrator

* dynamo config for docker

* docker endpoint for dynamo config throughout, working

* ignore python init

* remove enhanced table that was deprecated

* add curl to dockerfile

* docker compose includes ollama

* run delphi script checks for ollama avail, defaults to ll3.18b

* ollama generalization vs remote llm, working locally

* strip quotes

* claude generating report narratives.

* ensure datamapplot is in scope — docker fix

* run visualization after other steps, copy result out to disk

* info, not debug from boto

* debug model provider

* generate all topics for layer 0

* generate a static plot

* ignore png and svg image output from static plots

* Abstract docker-compose configuration.

* Integrate PCA/K-Means/Rep. pipeline.

* 702 color by consesus / divisive

* add 702 to run pipeline

* merge dynamo and docker take 1

* endpoint dynamo

* create table job

* little cli

* poller

* Implement persistent job queue with optimistic locking

- Change job table schema to use job_id as primary key
      - Add optimistic locking with version field
      - Ensure jobs are never deleted, only updated with new statuses
      - Implement strong consistency for critical operations
      - Fix DynamoDB compatibility issues with empty strings

* docs for system

* deprecated, now unified in top level create tables

* add job status cli command

* cli checks status checks status

* poller logging

* rename tables

* docs

* delphi cdk 1

* docker delphi

* add narrative to dockerfile

* run delphi with db fix, remove unsued file

* stubbing job hierarchy

* update job logs claude context

* example env for ollama

* log missing anthropic key

* Add automatic Ollama model setup and environment variable handling for jobs

* python error

* ignore node modeuls for node scripts in delphi

* deprecate delphi compose

* parametirize ollama model for topics

* complete db rename

* fix list convos in cli

* remove default model

* get rid of default model

* service url claude for client report

* try catch around conversation.py

* hello world move delphi to report client

* Load from DynamoDB.

* commentsReport route

* s3 and submit job from client

* no viz error message

* Load data properly from DynamoDB and check results are populated.

* guard llm names

* docs

* success showing report in commentsReport

* remove cleanup script

* add uuid

* node batch route

* Add PCAPlot component with ASCII-style symbols for high-density retro visualization

* don't pop browser

* batch reports endpoint

* batch reports endpoint

* skip static, don't pop browser

* don't pop browser

* batch debugging

* batch polling

* only run layer 0

* batch

* Improve batch processing with new job type architecture

This commit introduces a new approach to batch processing jobs with clearer job types:

1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py

The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* narrative batch

* create user fix, reset db script, commentsreport list

* Unified comment extremity calculation. Stored in DynamoDB.

* run delphy python conversion

* switch off sh to python

* call python job poller

* go to create narrative batch rather than narrative batch, explicit

* commentsReport linter

* tmp citations comment

* copy py into docker

* poller grab python

* wrap json parse for narrative

* update delphi for create_narrative_report batch

* render all cited comments

* make arch ready (#2014)

* make arch ready

* begin splitting up cdk code

* organize cdk

* cdk efs and permissions fixes

* docker fixes, add block storage

* remove default dynamo endpoint

* endpoint check fix

* use us-east-1

* remove more hardcoded values

* adjust delphi dockerfile

* fix up setupdynamo func

* fix reference

* add ssl mode hardcoded

* another place to add secure mode

* ssl fix

* update iam roles

* remove hardcoded minio from docker

* remove s3 check

* change s3 endpoint to none if missingf

* more s3 config

* fix indentation

* s3 logic fix

* indentation fix

* remove env check

* comment out s3 creds

* remove location constraint

* unique bucket name

* remove public read on aws

* fix comment

* endpoint urls in s3

* remove another dynamo default

* more endpoint config

* lint and test cleanup

* more lint fix

* defaults fixes

* tsc fix

* attempt free up disk space

* multi stage builds

* more space optimization

* more optimization

* debug

* remove commented out docker code, readme and makefile updates

* build fix local vs CI

* pass arg to ci

* add missing comma

* ssl mode fixes

* modify defaults and change bucket name

* side by side

* Mark jobs as completed.

* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py

* Improve nested log handling in job poller.

* add report bundles to proxy static files...

* s3 for delphi routes in node server

* batches running

* show batch time run job timestamp

* runs timestamps

* quiet batch check debug logging

* fix 700 s3 upload config

* hash delim

* show timestamp of report in ui

* logging

* no-op, 600 not used, but changes to topic naming

* topic naming logic and display

* batch uses new topic naming

* clean up topic name quotes

* happy memorial day bug squash

* no batch limit

* llm model as param

* concise logging

* add topic report route

* topicReport shows comments

* export report embeddable

* export report responsiveness

* deps update

* viz per layer

* comments report gets dropdown

* switch scan command for query command

* fix sorting

* we have two dropdowns we should unify later

* begin add GSI for perf improvements (#2029)

* begin add GSI for perf improvements

* refactor to avoid scan

* move config to outside of handler func

* replace another table scan

* clean up CLI dynamo

* fix 801

* cleaner batch jobs

* remove another table scan

* viz job client logic

---------

Co-authored-by: Colin Megill <[email protected]>

* actions fix

* fix reports gsi query

* removing zid from responses, client doesn't consume, running

* update docs after checsk

* viz zid

* batch report rid

* fix indentation issues

* integrate job id

* topicVizReport view

* fix reportsByJobId

* copy

* attempt jest test fixes

* add clear runner space to jest tests job

* resolve port conflict from pg

---------

Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>

* reports rendering with new batch format

* both commentsReport and topicsReport use both routes.

* narrative filtering

* componetize, and map jobs and sections

* default seelct consensus

* clean up logging

* reset single convo

* copywriting

* match viz job id with topics job id

* update cdk (#2037)

* use same access key name, simplify boto3 connection (#2038)

* use same access key name, simplify boto3 connection

* remove duplicates

* fix s3 url & py script (#2040)

* bad creds fix (#2042)

* bad creds fix

* clean up env

* remove infinite loop (#2044)

* switch to on demand capacity for some tables (#2046)

* switch to on demand capacity for some tables

* job poll updates

* poller updates final

* fix invalid url bug (#2048)

* claim job fix (#2050)

* fix ability to claim locked jobs (#2052)

* remove more bad dynamo urls (#2054)

* fix date sorting logic (#2056)

* poling simplification and lock cleaning (#2058)

* prevent subprocess timeouts in prod (#2060)

* increase timeout (#2062)

* bulk test ignore

* Te client side viz fixes (#2064)

* find best match in key

* add correct prefix

* rss feed route, and topics

* consensus feed rss prototype

* better filtering client side (#2067)

* fix key error (#2069)

* upgrade delphi images (#2071)

* Some delphi comments cleanup + JSON repairing (#2073)

* remove unnecessary comments

* json healing and comment removing

* Te topic map narrative report (#2076)

* begin add topicMapNarrativeReport

* clean up page

* Te delphi instance size config (#2077)

* respect instance sizes

* err fixes

* make comments more responsive (#2080)

* make comments more responsive

* move rawdataexport to bottom

* Te bulk statement upload (#2081)

* begin basic bulk upload capability

* save state

* minor fixes

* Te comment moderation 2.0 (#2084)

* begin mod capability

* update gemini api

* fix xml output

* lightening fast

* clean up and fine tuning

* don't analyze comment from mods

* fix lint

* test fix

* try test fixing

* add another mock

* try to make mock global

* try better mock path

* move setup file

* delete test for unused code path

* try more debugging

* attempt try catch

* remove console log

* add back deleted suite

* fix instance size variable so large jobs are processed (#2087)

* Te job poller size fix (#2089)

* fix instance size variable so large jobs are processed

* add to docker

* Te job poller size fix (#2091)

* fix instance size variable so large jobs are processed

* add to docker

* update install script

* Te job poller size fix (#2093)

* fix instance size variable so large jobs are processed

* add to docker

* update install script

* replace all instances of old var

* Full OIDC + JWT Integration (#2024)

Full Integration of OIDC authentication system.
Deprecation of legacy cookie-based auth.
Upgrade client-admin app.
OIDC Simulator for dev/test environments.

* add missing actions vars

* Fix comment filtering bug (#2111)

* change from tmp (#2112)

* add reset script

* update email sending

* revert changes but with more explicit error handling

* Te client participation 2 (#2098)

* add astro version of client-participation

* minor style updates

* begin moving over translations

* port over translations

* ssr

* Use real convo data

* dynamically update title

* more translations

* wire in voting capability

* get voting working

* cors

* type check

* impliment convo closed status

* admin mod, prio stub

* hclust route checkpoitn

* labels working

* remove logging, get towards hclust

* hclust explration 22

* contour plot

* contours

* prio take 1

* distances

* remove dead 600 file

* add more defense for small models

* topic prioritize mock

* webpack build size

* topic sections

* merge fix

* add simple route

* Restructure priority calculation to use group-based extremity

- Remove priority calculation from math pipeline (conversation.py)
  * Delete _importance_metric and _priority_metric methods
  * Remove priority computation from recompute() method
  * Math pipeline now focuses on PCA, clustering, and representativeness

- Add dedicated priority calculation script (502_calculate_priorities.py)
  * Implements PriorityCalculator class with group-based extremity
  * Matches Clojure priority formula: (importance * scaling_factor)^2
  * Retrieves extremity values from Delphi_CommentExtremity table
  * Updates priorities in Delphi_CommentRouting table

- Update pipeline execution order (run_delphi.py, run_delphi.sh)
  * Math pipeline → UMAP pipeline → Extremity calculation → Priority calculation
  * Ensures priorities use group-based extremity instead of PCA-based
  * Maintains separation of concerns between mathematical and priority calculations

This fixes the priority calculation bug where all priorities were 0 due to
missing extremity values, and implements proper group-based extremity usage
as requested for the Pakistan conversation analysis.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Refactor priority calculation: separate formulas from service logic

- Extract priority formulas into polismath/conversation/priority.py
  * Create PriorityCalculator class with static methods for core formulas
  * Port importance_metric and priority_metric from Clojure implementation
  * Add convenience methods: calculate_comment_priority, validate_inputs, explain_priority
  * Pure mathematical logic with no I/O dependencies for better testability

- Refactor umap_narrative/502_calculate_priorities.py to use extracted formulas
  * Rename class from PriorityCalculator to PriorityService (clearer distinction)
  * Import and use PriorityCalculator.calculate_comment_priority()
  * Remove duplicate formula implementations (38 lines removed)
  * Focus service on DynamoDB operations and data orchestration

Benefits:
- Better separation of concerns: formulas vs data processing
- Improved testability: mathematical logic can be unit tested independently
- Enhanced reusability: priority formulas can be used in other contexts
- Cleaner maintainability: formula changes only need to happen in one place

Tested successfully with conversation 36324:
- 807 comments processed
- Priority statistics: min=0, max=3, avg=2.36
- All priorities calculated using group-based extremity values

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Revert "Refactor priority calculation: separate formulas from service logic"

This reverts commit cfd829d.

* topic agenda collapse prototype tetris-y

* topic agenda setting demo

* begin basic bulk upload capability

* save state

* split by distance doesn't owrk

* sentence transformer model

* componetize

* drive detail

* more specifics

* log archetypal comments

* dynamo table and node services

* Don't crash when we have not votes.

* suppress logging

* differnet approach for prepended labesl

* Te arch updates topicmod (#2086)

* add polis mod access and remove table scans

* add comments back

* more comment fixes

* add footer links

* begin dockerization

* dockerize and new route structure

* wire in xid

* get alpha embed working

* prune comments

* connect comment form

* translations fixes, hook up subscribe form

* css fixes

* fix env vars

* Colinmegill/topic prio (#2103)

* showing topics in participant view

* logo color / p slightly more minimal change

---------

Co-authored-by: Colin Megill <[email protected]>

* perf adjustments

* remove console log

* minor fixes

* compilation fixes

* more auth

* get auth working

* return vote response

* participation alpha qa and better topic loading

* add hide header ability

* update headerless

* turn off eslint for now

* prep alpha for prod deploy

---------

Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>

* update compose-test

* fix test compose network name

* compose test cert volume

* update test action

* add cpa to test file

* gh actions copy certs for tests

---------

Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>

* Cleanup Pakistan workflow and write to DynamoDB.

* Add Pakistan test for current implementation.

* Add requirements.

* umap narrative

* move docs.

* nuke test gpu direction

* add evoc to gitigonre

a

* merge req txt

* delete other req txt

* remove notebooks

* unified create dynamo tables

* add fastapi to reqs

* add missing group votes initialization

* error handinling in db connect

* experimental dockerization

* concise readme

* HAS TO BE ROOT USER!!!

* run_delphi talks to dynamo internally :)

* all dependencies in req.txt, and c++ in dockerfile.

* run command

* let's run the umap pipeline directly, wait on orchestrator

* dynamo config for docker

* docker endpoint for dynamo config throughout, working

* ignore python init

* remove enhanced table that was deprecated

* add curl to dockerfile

* docker compose includes ollama

* run delphi script checks for ollama avail, defaults to ll3.18b

* ollama generalization vs remote llm, working locally

* strip quotes

* claude generating report narratives.

* ensure datamapplot is in scope — docker fix

* run visualization after other steps, copy result out to disk

* info, not debug from boto

* debug model provider

* generate all topics for layer 0

* generate a static plot

* ignore png and svg image output from static plots

* Abstract docker-compose configuration.

* Integrate PCA/K-Means/Rep. pipeline.

* 702 color by consesus / divisive

* add 702 to run pipeline

* merge dynamo and docker take 1

* endpoint dynamo

* create table job

* little cli

* poller

* Implement persistent job queue with optimistic locking

- Change job table schema to use job_id as primary key
      - Add optimistic locking with version field
      - Ensure jobs are never deleted, only updated with new statuses
      - Implement strong consistency for critical operations
      - Fix DynamoDB compatibility issues with empty strings

* docs for system

* deprecated, now unified in top level create tables

* add job status cli command

* cli checks status checks status

* poller logging

* rename tables

* docs

* delphi cdk 1

* docker delphi

* add narrative to dockerfile

* run delphi with db fix, remove unsued file

* stubbing job hierarchy

* update job logs claude context

* example env for ollama

* log missing anthropic key

* Add automatic Ollama model setup and environment variable handling for jobs

* python error

* ignore node modeuls for node scripts in delphi

* deprecate delphi compose

* parametirize ollama model for topics

* complete db rename

* fix list convos in cli

* remove default model

* get rid of default model

* service url claude for client report

* try catch around conversation.py

* hello world move delphi to report client

* Load from DynamoDB.

* commentsReport route

* s3 and submit job from client

* no viz error message

* Load data properly from DynamoDB and check results are populated.

* guard llm names

* docs

* success showing report in commentsReport

* remove cleanup script

* add uuid

* node batch route

* Add PCAPlot component with ASCII-style symbols for high-density retro visualization

* don't pop browser

* batch reports endpoint

* batch reports endpoint

* skip static, don't pop browser

* don't pop browser

* batch debugging

* batch polling

* only run layer 0

* batch

* Improve batch processing with new job type architecture

This commit introduces a new approach to batch processing jobs with clearer job types:

1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py

The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* narrative batch

* create user fix, reset db script, commentsreport list

* Unified comment extremity calculation. Stored in DynamoDB.

* run delphy python conversion

* switch off sh to python

* call python job poller

* go to create narrative batch rather than narrative batch, explicit

* commentsReport linter

* tmp citations comment

* copy py into docker

* poller grab python

* wrap json parse for narrative

* update delphi for create_narrative_report batch

* render all cited comments

* make arch ready (#2014)

* make arch ready

* begin splitting up cdk code

* organize cdk

* cdk efs and permissions fixes

* docker fixes, add block storage

* remove default dynamo endpoint

* endpoint check fix

* use us-east-1

* remove more hardcoded values

* adjust delphi dockerfile

* fix up setupdynamo func

* fix reference

* add ssl mode hardcoded

* another place to add secure mode

* ssl fix

* update iam roles

* remove hardcoded minio from docker

* remove s3 check

* change s3 endpoint to none if missingf

* more s3 config

* fix indentation

* s3 logic fix

* indentation fix

* remove env check

* comment out s3 creds

* remove location constraint

* unique bucket name

* remove public read on aws

* fix comment

* endpoint urls in s3

* remove another dynamo default

* more endpoint config

* lint and test cleanup

* more lint fix

* defaults fixes

* tsc fix

* attempt free up disk space

* multi stage builds

* more space optimization

* more optimization

* debug

* remove commented out docker code, readme and makefile updates

* build fix local vs CI

* pass arg to ci

* add missing comma

* ssl mode fixes

* modify defaults and change bucket name

* side by side

* Mark jobs as completed.

* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py

* Improve nested log handling in job poller.

* add report bundles to proxy static files...

* s3 for delphi routes in node server

* batches running

* show batch time run job timestamp

* runs timestamps

* quiet batch check debug logging

* fix 700 s3 upload config

* hash delim

* show timestamp of report in ui

* logging

* no-op, 600 not used, but changes to topic naming

* topic naming logic and display

* batch uses new topic naming

* clean up topic name quotes

* happy memorial day bug squash

* no batch limit

* llm model as param

* concise logging

* add topic report route

* topicReport shows comments

* export report embeddable

* export report responsiveness

* deps update

* viz per layer

* comments report gets dropdown

* switch scan command for query command

* fix sorting

* we have two dropdowns we should unify later

* begin add GSI for perf improvements (#2029)

* begin add GSI for perf improvements

* refactor to avoid scan

* move config to outside of handler func

* replace another table scan

* clean up CLI dynamo

* fix 801

* cleaner batch jobs

* remove another table scan

* viz job client logic

---------

Co-authored-by: Colin Megill <[email protected]>

* actions fix

* fix reports gsi query

* removing zid from responses, client doesn't consume, running

* update docs after checsk

* viz zid

* batch report rid

* fix indentation issues

* integrate job id

* topicVizReport view

* fix reportsByJobId

* copy

* get all layers, get consensus...

* Colinmegill/node delphi (#2027)

* batch debugging

* batch polling

* only run layer 0

* batch

* Improve batch processing with new job type architecture

This commit introduces a new approach to batch processing jobs with clearer job types:

1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py

The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* narrative batch

* create user fix, reset db script, commentsreport list

* Unified comment extremity calculation. Stored in DynamoDB.

* run delphy python conversion

* switch off sh to python

* call python job poller

* go to create narrative batch rather than narrative batch, explicit

* commentsReport linter

* tmp citations comment

* copy py into docker

* poller grab python

* wrap json parse for narrative

* update delphi for create_narrative_report batch

* render all cited comments

* make arch ready (#2014)

* make arch ready

* begin splitting up cdk code

* organize cdk

* cdk efs and permissions fixes

* docker fixes, add block storage

* remove default dynamo endpoint

* endpoint check fix

* use us-east-1

* remove more hardcoded values

* adjust delphi dockerfile

* fix up setupdynamo func

* fix reference

* add ssl mode hardcoded

* another place to add secure mode

* ssl fix

* update iam roles

* remove hardcoded minio from docker

* remove s3 check

* change s3 endpoint to none if missingf

* more s3 config

* fix indentation

* s3 logic fix

* indentation fix

* remove env check

* comment out s3 creds

* remove location constraint

* unique bucket name

* remove public read on aws

* fix comment

* endpoint urls in s3

* remove another dynamo default

* more endpoint config

* lint and test cleanup

* more lint fix

* defaults fixes

* tsc fix

* attempt free up disk space

* multi stage builds

* more space optimization

* more optimization

* debug

* remove commented out docker code, readme and makefile updates

* build fix local vs CI

* pass arg to ci

* add missing comma

* ssl mode fixes

* Mark jobs as completed.

* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py

* modify defaults and change bucket name

* side by side

* Improve nested log handling in job poller.

* add report bundles to proxy static files...

* s3 for delphi routes in node server

* batches running

* show batch time run job timestamp

* runs timestamps

* quiet batch check debug logging

* fix 700 s3 upload config

* hash delim

* show timestamp of report in ui

* logging

* no-op, 600 not used, but changes to topic naming

* topic naming logic and display

* batch uses new topic naming

* clean up topic name quotes

* happy memorial day bug squash

* no batch limit

* llm model as param

* concise logging

* add topic report route

* topicReport shows comments

* export report embeddable

* export report responsiveness

* remove .sh references

* bump to 16 gigs default, and numba jit

* remove shell script rferences from dockerfile

* remove sh commands

* bump python deps

* run, and show, all visualizations in comments Report

* claude code reference

* notebooks

* python tests against real data, compare with clojure

* notebooks

* vw analysis output

* vw correlation plot

* vw output

* vw notebook

* Revert "vw notebook"

This reverts commit 7548529.

* biodiversity notebook

* pca validation notebook

* pca validation notebook fix

* add ignore, add analysis check.

* gpu test

* benchmark pngs

* move math/python_conversion to delphi dir

* setup docker for delphi

* update path naming python_converstion -> delph

* corrections to postgres schema

* Add PostgreSQL integration tests for biodiversity dataset

* Add AWS Lambda function for biodiversity dataset processing

* Add optimized math functionality to delphi branch and fix PostgreSQL integration tests

* Remove incorrectly added math/python_conversion files - all code should be in delphi/polismath

* Cleanup Pakistan workflow and write to DynamoDB.

* Add Pakistan test for current implementation.

* Add requirements.

* umap narrative

* move docs.

* nuke test gpu direction

* add evoc to gitigonre

a

* merge req txt

* delete other req txt

* remove notebooks

* unified create dynamo tables

* add fastapi to reqs

* add missing group votes initialization

* error handinling in db connect

* experimental dockerization

* concise readme

* HAS TO BE ROOT USER!!!

* run_delphi talks to dynamo internally :)

* all dependencies in req.txt, and c++ in dockerfile.

* run command

* let's run the umap pipeline directly, wait on orchestrator

* dynamo config for docker

* docker endpoint for dynamo config throughout, working

* ignore python init

* remove enhanced table that was deprecated

* add curl to dockerfile

* docker compose includes ollama

* run delphi script checks for ollama avail, defaults to ll3.18b

* ollama generalization vs remote llm, working locally

* strip quotes

* claude generating report narratives.

* ensure datamapplot is in scope — docker fix

* run visualization after other steps, copy result out to disk

* info, not debug from boto

* debug model provider

* generate all topics for layer 0

* generate a static plot

* ignore png and svg image output from static plots

* Abstract docker-compose configuration.

* Integrate PCA/K-Means/Rep. pipeline.

* 702 color by consesus / divisive

* add 702 to run pipeline

* merge dynamo and docker take 1

* endpoint dynamo

* create table job

* little cli

* poller

* Implement persistent job queue with optimistic locking

- Change job table schema to use job_id as primary key
      - Add optimistic locking with version field
      - Ensure jobs are never deleted, only updated with new statuses
      - Implement strong consistency for critical operations
      - Fix DynamoDB compatibility issues with empty strings

* docs for system

* deprecated, now unified in top level create tables

* add job status cli command

* cli checks status checks status

* poller logging

* rename tables

* docs

* delphi cdk 1

* docker delphi

* add narrative to dockerfile

* run delphi with db fix, remove unsued file

* stubbing job hierarchy

* update job logs claude context

* example env for ollama

* log missing anthropic key

* Add automatic Ollama model setup and environment variable handling for jobs

* python error

* ignore node modeuls for node scripts in delphi

* deprecate delphi compose

* parametirize ollama model for topics

* complete db rename

* fix list convos in cli

* remove default model

* get rid of default model

* service url claude for client report

* try catch around conversation.py

* hello world move delphi to report client

* Load from DynamoDB.

* commentsReport route

* s3 and submit job from client

* no viz error message

* Load data properly from DynamoDB and check results are populated.

* guard llm names

* docs

* success showing report in commentsReport

* remove cleanup script

* add uuid

* node batch route

* Add PCAPlot component with ASCII-style symbols for high-density retro visualization

* don't pop browser

* batch reports endpoint

* batch reports endpoint

* skip static, don't pop browser

* don't pop browser

* batch debugging

* batch polling

* only run layer 0

* batch

* Improve batch processing with new job type architecture

This commit introduces a new approach to batch processing jobs with clearer job types:

1. CREATE_NARRATIVE_BATCH: Jobs that create batches via 801_narrative_report_batch.py
2. AWAITING_NARRATIVE_BATCH: Jobs that check batch status via 803_check_batch_status.py

The change fixes a critical issue where jobs were running the wrong script, and
adds proper handling of DynamoDB reserved keywords like 'status'. This state
machine design is now documented in docs/JOB_STATE_MACHINE_DESIGN.md.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* narrative batch

* create user fix, reset db script, commentsreport list

* Unified comment extremity calculation. Stored in DynamoDB.

* run delphy python conversion

* switch off sh to python

* call python job poller

* go to create narrative batch rather than narrative batch, explicit

* commentsReport linter

* tmp citations comment

* copy py into docker

* poller grab python

* wrap json parse for narrative

* update delphi for create_narrative_report batch

* render all cited comments

* make arch ready (#2014)

* make arch ready

* begin splitting up cdk code

* organize cdk

* cdk efs and permissions fixes

* docker fixes, add block storage

* remove default dynamo endpoint

* endpoint check fix

* use us-east-1

* remove more hardcoded values

* adjust delphi dockerfile

* fix up setupdynamo func

* fix reference

* add ssl mode hardcoded

* another place to add secure mode

* ssl fix

* update iam roles

* remove hardcoded minio from docker

* remove s3 check

* change s3 endpoint to none if missingf

* more s3 config

* fix indentation

* s3 logic fix

* indentation fix

* remove env check

* comment out s3 creds

* remove location constraint

* unique bucket name

* remove public read on aws

* fix comment

* endpoint urls in s3

* remove another dynamo default

* more endpoint config

* lint and test cleanup

* more lint fix

* defaults fixes

* tsc fix

* attempt free up disk space

* multi stage builds

* more space optimization

* more optimization

* debug

* remove commented out docker code, readme and makefile updates

* build fix local vs CI

* pass arg to ci

* add missing comma

* ssl mode fixes

* modify defaults and change bucket name

* side by side

* Mark jobs as completed.

* Add AWAITING_NARRATIVE_BATCH job type to delphi_cli.py and fix syntax error in job_poller.py

* Improve nested log handling in job poller.

* add report bundles to proxy static files...

* s3 for delphi routes in node server

* batches running

* show batch time run job timestamp

* runs timestamps

* quiet batch check debug logging

* fix 700 s3 upload config

* hash delim

* show timestamp of report in ui

* logging

* no-op, 600 not used, but changes to topic naming

* topic naming logic and display

* batch uses new topic naming

* clean up topic name quotes

* happy memorial day bug squash

* no batch limit

* llm model as param

* concise logging

* add topic report route

* topicReport shows comments

* export report embeddable

* export report responsiveness

* deps update

* viz per layer

* comments report gets dropdown

* switch scan command for query command

* fix sorting

* we have two dropdowns we should unify later

* begin add GSI for perf improvements (#2029)

* begin add GSI for perf improvements

* refactor to avoid scan

* move config to outside of handler func

* replace another table scan

* clean up CLI dynamo

* fix 801

* cleaner batch jobs

* remove another table scan

* viz job client logic

---------

Co-authored-by: Colin Megill <[email protected]>

* actions fix

* fix reports gsi query

* removing zid from responses, client doesn't consume, running

* update docs after checsk

* viz zid

* batch report rid

* fix indentation issues

* integrate job id

* topicVizReport view

* fix reportsByJobId

* copy

* attempt jest test fixes

* add clear runner space to jest tests job

* resolve port conflict from pg

---------

Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>

* reports rendering with new batch format

* both commentsReport and topicsReport use both routes.

* narrative filtering

* componetize, and map jobs and sections

* default seelct consensus

* clean up logging

* reset single convo

* copywriting

* match viz job id with topics job id

* update cdk (#2037)

* use same access key name, simplify boto3 connection (#2038)

* use same access key name, simplify boto3 connection

* remove duplicates

* fix s3 url & py script (#2040)

* bad creds fix (#2042)

* bad creds fix

* clean up env

* remove infinite loop (#2044)

* switch to on demand capacity for some tables (#2046)

* switch to on demand capacity for some tables

* job poll updates

* poller updates final

* fix invalid url bug (#2048)

* claim job fix (#2050)

* fix ability to claim locked jobs (#2052)

* remove more bad dynamo urls (#2054)

* fix date sorting logic (#2056)

* poling simplification and lock cleaning (#2058)

* prevent subprocess timeouts in prod (#2060)

* increase timeout (#2062)

* bulk test ignore

* Te client side viz fixes (#2064)

* find best match in key

* add correct prefix

* rss feed route, and topics

* consensus feed rss prototype

* better filtering client side (#2067)

* fix key error (#2069)

* upgrade delphi images (#2071)

* Some delphi comments cleanup + JSON repairing (#2073)

* remove unnecessary comments

* json healing and comment removing

* Te topic map narrative report (#2076)

* begin add topicMapNarrativeReport

* clean up page

* Te delphi instance size config (#2077)

* respect instance sizes

* err fixes

* make comments more responsive (#2080)

* make comments more responsive

* move rawdataexport to bottom

* Te bulk statement upload (#2081)

* begin basic bulk upload capability

* save state

* minor fixes

* Te comment moderation 2.0 (#2084)

* begin mod capability

* update gemini api

* fix xml output

* lightening fast

* clean up and fine tuning

* don't analyze comment from mods

* fix lint

* test fix

* try test fixing

* add another mock

* try to make mock global

* try better mock path

* move setup file

* delete test for unused code path

* try more debugging

* attempt try catch

* remove console log

* add back deleted suite

* fix instance size variable so large jobs are processed (#2087)

* Te job poller size fix (#2089)

* fix instance size variable so large jobs are processed

* add to docker

* Te job poller size fix (#2091)

* fix instance size variable so large jobs are processed

* add to docker

* update install script

* Te job poller size fix (#2093)

* fix instance size variable so large jobs are processed

* add to docker

* update install script

* replace all instances of old var

* Full OIDC + JWT Integration (#2024)

Full Integration of OIDC authentication system.
Deprecation of legacy cookie-based auth.
Upgrade client-admin app.
OIDC Simulator for dev/test environments.

* add missing actions vars

* Fix comment filtering bug (#2111)

* change from tmp (#2112)

* add reset script

* update email sending

* revert changes but with more explicit error handling

* Te client participation 2 (#2098)

* add astro version of client-participation

* minor style updates

* begin moving over translations

* port over translations

* ssr

* Use real convo data

* dynamically update title

* more translations

* wire in voting capability

* get voting working

* cors

* type check

* impliment convo closed status

* admin mod, prio stub

* hclust route checkpoitn

* labels working

* remove logging, get towards hclust

* hclust explration 22

* contour plot

* contours

* prio take 1

* distances

* remove dead 600 file

* add more defense for small models

* topic prioritize mock

* webpack build size

* topic sections

* merge fix

* add simple route

* Restructure priority calculation to use group-based extremity

- Remove priority calculation from math pipeline (conversation.py)
  * Delete _importance_metric and _priority_metric methods
  * Remove priority computation from recompute() method
  * Math pipeline now focuses on PCA, clustering, and representativeness

- Add dedicated priority calculation script (502_calculate_priorities.py)
  * Implements PriorityCalculator class with group-based extremity
  * Matches Clojure priority formula: (importance * scaling_factor)^2
  * Retrieves extremity values from Delphi_CommentExtremity table
  * Updates priorities in Delphi_CommentRouting table

- Update pipeline execution order (run_delphi.py, run_delphi.sh)
  * Math pipeline → UMAP pipeline → Extremity calculation → Priority calculation
  * Ensures priorities use group-based extremity instead of PCA-based
  * Maintains separation of concerns between mathematical and priority calculations

This fixes the priority calculation bug where all priorities were 0 due to
missing extremity values, and implements proper group-based extremity usage
as requested for the Pakistan conversation analysis.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Refactor priority calculation: separate formulas from service logic

- Extract priority formulas into polismath/conversation/priority.py
  * Create PriorityCalculator class with static methods for core formulas
  * Port importance_metric and priority_metric from Clojure implementation
  * Add convenience methods: calculate_comment_priority, validate_inputs, explain_priority
  * Pure mathematical logic with no I/O dependencies for better testability

- Refactor umap_narrative/502_calculate_priorities.py to use extracted formulas
  * Rename class from PriorityCalculator to PriorityService (clearer distinction)
  * Import and use PriorityCalculator.calculate_comment_priority()
  * Remove duplicate formula implementations (38 lines removed)
  * Focus service on DynamoDB operations and data orchestration

Benefits:
- Better separation of concerns: formulas vs data processing
- Improved testability: mathematical logic can be unit tested independently
- Enhanced reusability: priority formulas can be used in other contexts
- Cleaner maintainability: formula changes only need to happen in one place

Tested successfully with conversation 36324:
- 807 comments processed
- Priority statistics: min=0, max=3, avg=2.36
- All priorities calculated using group-based extremity values

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Revert "Refactor priority calculation: separate formulas from service logic"

This reverts commit cfd829d.

* topic agenda collapse prototype tetris-y

* topic agenda setting demo

* begin basic bulk upload capability

* save state

* split by distance doesn't owrk

* sentence transformer model

* componetize

* drive detail

* more specifics

* log archetypal comments

* dynamo table and node services

* Don't crash when we have not votes.

* suppress logging

* differnet approach for prepended labesl

* Te arch updates topicmod (#2086)

* add polis mod access and remove table scans

* add comments back

* more comment fixes

* add footer links

* begin dockerization

* dockerize and new route structure

* wire in xid

* get alpha embed working

* prune comments

* connect comment form

* translations fixes, hook up subscribe form

* css fixes

* fix env vars

* Colinmegill/topic prio (#2103)

* showing topics in participant view

* logo color / p slightly more minimal change

---------

Co-authored-by: Colin Megill <[email protected]>

* perf adjustments

* remove console log

* minor fixes

* compilation fixes

* more auth

* get auth working

* return vote response

* participation alpha qa and better topic loading

* add hide header ability

* update headerless

* turn off eslint for now

* prep alpha for prod deploy

---------

Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>

* update compose-test

* fix test compose network name

* compose test cert volume

* update test action

* add cpa to test file

* gh actions copy certs for tests

* fix token in header

* update token logic

---------

Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Colin Megill <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Bennie Rosas <[email protected]>

PROD DEPLOY 4 8/7

…eStatement

Bumps [astro](https://github.com/withastro/astro/tree/HEAD/packages/astro) from 5.11.0 to 5.12.8.
- [Release notes](https://github.com/withastro/astro/releases)
- [Changelog](https://github.com/withastro/astro/blob/main/packages/astro/CHANGELOG.md)
- [Commits](https://github.com/withastro/astro/commits/[email protected]/packages/astro)

---
updated-dependencies:
- dependency-name: astro
  dependency-version: 5.12.8
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* token cleanup and load topic fix (#2123)

* Topic Selection and New Comment Routing (#2117)

# Configurable Topic-Informed Comment Routing for Alpha Client (and more)

## Client Participation Alpha

- centralize JWT handling
- convert net.js to typescript
- fetch NextComment upon hydration for personalization (ie don't show voted on comment)

## Server

- dedicated NextComment module
- checks for TOPICAL_COMMENT_RATIO (float)
- if ^ is set, blends NextComment selection between Topic-Informed comment pool and traditional prioritization
- graceful handling of empty-ish pca objects when there are comments but zero votes
- include DynamoDB helpers in integration tests
- integration tests for NextComment routing

## E2E

- test comment csv upload
- test report generation and viewing for "empty-ish" pca with comments but zero votes

## Client Report

- graceful handling of empty-ish pca objects (has comments, zero votes)

## Math

- enhanced Postgres connection pooling
- reduced risk of exhausting postgres connections and crashing
- reduced risk of "zombie" postgres connections that don't get cleaned up (mainly in test runs)

## Postgres

- *Migration* 000012 creates topic_agenda_selections table
- Improved stability in test & dev docker setups

## Misc

- consolidated Authentication docs
- improved testing stability overall
- update example.env files

* Bump astro from 5.11.0 to 5.12.8 in /client-participation-alpha (#2119)

Bumps [astro](https://github.com/withastro/astro/tree/HEAD/packages/astro) from 5.11.0 to 5.12.8.
- [Release notes](https://github.com/withastro/astro/releases)
- [Changelog](https://github.com/withastro/astro/blob/main/packages/astro/CHANGELOG.md)
- [Commits](https://github.com/withastro/astro/commits/[email protected]/packages/astro)

---
updated-dependencies:
- dependency-name: astro
  dependency-version: 5.12.8
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Tim <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* e2e dont warn on cy.wait usage

* set POLIS_TOPICAL_RATIO in jest global setup

* updated test setup

* try without snake oil

* dynamo use host.docker.internal

* ci-safe file handling in cypress

* jest ci use dynamodb at localhost

* dynamo config for ci jest server tests

* add astro version of client-participation

* minor style updates

* begin moving over translations

* port over translations

* ssr

* Use real convo data

* dynamically update title

* more translations

* wire in voting capability

* get voting working

* cors

* type check

* impliment convo closed status

* admin mod, prio stub

* hclust route checkpoitn

* labels working

* remove logging, get towards hclust

* hclust explration 22

* contour plot

* contours

* prio take 1

* distances

* remove dead 600 file

* add more defense for small models

* topic prioritize mock

* webpack build size

* topic sections

* merge fix

* add simple route

* Restructure priority calculation to use group-based extremity

- Remove priority calculation from math pipeline (conversation.py)
  * Delete _importance_metric and _priority_metric methods
  * Remove priority computation from recompute() method
  * Math pipeline now focuses on PCA, clustering, and representativeness

- Add dedicated priority calculation script (502_calculate_priorities.py)
  * Implements PriorityCalculator class with group-based extremity
  * Matches Clojure priority formula: (importance * scaling_factor)^2
  * Retrieves extremity values from Delphi_CommentExtremity table
  * Updates priorities in Delphi_CommentRouting table

- Update pipeline execution order (run_delphi.py, run_delphi.sh)
  * Math pipeline → UMAP pipeline → Extremity calculation → Priority calculation
  * Ensures priorities use group-based extremity instead of PCA-based
  * Maintains separation of concerns between mathematical and priority calculations

This fixes the priority calculation bug where all priorities were 0 due to
missing extremity values, and implements proper group-based extremity usage
as requested for the Pakistan conversation analysis.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Refactor priority calculation: separate formulas from service logic

- Extract priority formulas into polismath/conversation/priority.py
  * Create PriorityCalculator class with static methods for core formulas
  * Port importance_metric and priority_metric from Clojure implementation
  * Add convenience methods: calculate_comment_priority, validate_inputs, explain_priority
  * Pure mathematical logic with no I/O dependencies for better testability

- Refactor umap_narrative/502_calculate_priorities.py to use extracted formulas
  * Rename class from PriorityCalculator to PriorityService (clearer distinction)
  * Import and use PriorityCalculator.calculate_comment_priority()
  * Remove duplicate formula implementations (38 lines removed)
  * Focus service on DynamoDB operations and data orchestration

Benefits:
- Better separation of concerns: formulas vs data processing
- Improved testability: mathematical logic can be unit tested independently
- Enhanced reusability: priority formulas can be used in other contexts
- Cleaner maintainability: formula changes only need to happen in one place

Tested successfully with conversation 36324:
- 807 comments processed
- Priority statistics: min=0, max=3, avg=2.36
- All priorities calculated using group-based extremity values

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* Revert "Refactor priority calculation: separate formulas from service logic"

This reverts commit cfd829d.

* topic agenda collapse prototype tetris-y

* topic agenda setting demo

* begin basic bulk upload capability

* save state

* split by distance doesn't owrk

* sentence transformer model

* componetize

* drive detail

* more specifics

* log archetypal comments

* dynamo table and node services

* Don't crash when we have not votes.

* suppress logging

* differnet approach for prepended labesl

* Te arch updates topicmod (#2086)

* add polis mod access and remove table scans

* add comments back

* more comment fixes

* add footer links

* begin dockerization

* dockerize and new route structure

* wire in xid

* get alpha embed working

* prune comments

* connect comment form

* translations fixes, hook up subscribe form

* css fixes

* fix env vars

* Colinmegill/topic prio (#2103)

* showing topics in participant view

* logo color / p slightly more minimal change

---------

Co-authored-by: Colin Megill <[email protected]>

* perf adjustments

* remove console log

* minor fixes

* compilation fixes

* more auth

* get auth working

* return vote response

* participation alpha qa and better topic loading

* add hide header ability

* update headerless

* turn off eslint for now

* clean up delphi root.

* repair toipcMapNarrativeReport import

* topic stats rendering

* reset conversation script

* looking at cross topic gic measures

* reset jobs uitl

* reset batch util

* rename notebooks folder

* move markdown files to docs

* batch complete job

* revert dynamo scan

* show button on low divisive statemtns

* collective statements render

* table sort

* plotly scatterplot topic metrics

* update prompt

* avg group aware consensus

* blazing fast, great.

* green yellow red

* all comments plot

* sort!

* collective statement modal

* refactor componetize

* three modals.

* modals.

* topic page.

* topic page

* typography

* button cleanup

* button color

* disable topic mod link for now

* topic prioritize delete prototypte

* remove topicAgenda since now copied to ptpt alpha

* put back participnat topic prioritize

* lint hierarchy

* Normalized group aware consensus calculation

* thresholding .05 [email protected]

* url format, prompt tweak, copywrite,

* broken collective statement carousel alpha

* collective statemetn carousel

* carousel

* embed logic

* clean up collective statmetns before rerun

* only show most recent version in carousel

* Merge branch 'edge' into colinmegill/collectiveStatement

* fix stray line in app.js

* delete duplicate script

* update to query command, delete unused files

* make PCA optional

* don't load pca immediately

* linting fixes

---------

Co-authored-by: tevko <[email protected]>
Co-authored-by: Christian Weilbach <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Tim <[email protected]>