-
Notifications
You must be signed in to change notification settings - Fork 12
Adding Static Model Outputs to MaaS
This page describes how to a new “static” model to MaaS. This is just a model that we do not execute. Instead, we obtain its output, normalize it, store it, and allow a user to query for it through the /run_model endpoint.
There are currently two “models” that have static outputs:
First, you need to generate a metadata YAML file that describes the model. It should be stored in /metadata/models. Here is an example of this.
These YAML files are read into MaaS each time MaaS is started/restarted and are stored in Redis. The information is made available through various endpoints (e.g. /model_info/)
Model output will either be given to us, or in the case of WorldPop we need to scrape it/download it. We should start with the African continent
A model run (in the case of WorldPop) would just be each file.
For WorldPop, there are no obvious “tunable” parameters. However for other models this may not be true. For LPJmL Yield Anomalies, each file represents a run given a certain set of parameters (nitrogen level, crop, irrigation level, etc).
If for example, WorldPop provided 3 estimates per year (high, medium, low) then perhaps a parameter would be “Projection Level” or something like that. It doesn’t look like that is the case though.
Each file must be processed as though it were a run. You can see how this is accomplished for Atlas AI models or for LPJmL Yield Anomalies
Currently the two key activities are split out into separate files:
-
{model_name}_data.pyadds the run objects to Redis and pushes the files to S3. -
{model_name}_processing.pyconverts the rasters to points and uploads these to the Postgres database
You should spin up local Redis/Postgres with docker-compose up -d from the root directory of the project for development purposes.
You’ll also need to ensure that you set up a working installation of GDAL.
Eventually the Redis/Postgres instances may be combined; Postgres is new and Redis is legacy. Redis mostly feeds the API, but eventually we may just use the information in Postgres for this.
Probably the trickiest part of this process is ensuring that the raster conversion to point data succeeds and that you specify the appropriate metadata and output data for each run and each row (point) from each run. For reference, the SQL Alchemy database schema is stored here
run_label and run_description are nice to haves but very useful. For example, for World Pop a run_label might be something like “World Pop Ethiopia population estimate for 2020”. run_description could be a bit more verbose if need be but is something you’re less likely to use.
You’ll definitely want to ensure that the feature_name is set correctly (e.g. population per 1km, etc).
@brandomr can provide the Atlas AI and LPJmL Yield Anomalies data sets for testing if helpful. Getting one or two of those files ingested into your local DB and Redis would be a good starting point to making sure things are working as expected.