diff --git a/_data/docs.yml b/_data/docs.yml index 3b4fe12..feac8ef 100755 --- a/_data/docs.yml +++ b/_data/docs.yml @@ -29,7 +29,7 @@ - title: Plugins path: available-plugins/plugins -- title: Usage and Specifications +- title: Advanced usage path: usage/koop-core docs: - title: Koop Core @@ -41,4 +41,16 @@ - title: Output Spec path: usage/output - title: Authorization Spec - path: usage/authorization \ No newline at end of file + path: usage/authorization + +- title: Plugin Development + path: development/provider + docs: + - title: Provider Spec + path: development/provider + - title: Cache Spec + path: development/cache + - title: Output Spec + path: development/output + - title: Authorization Spec + path: development/authorization \ No newline at end of file diff --git a/_docs/basics/overview.md b/_docs/basics/overview.md index ad04d06..d2a9845 100644 --- a/_docs/basics/overview.md +++ b/_docs/basics/overview.md @@ -4,18 +4,16 @@ permalink: /docs/basics/overview --- A Koop instance usually includes the following components: [core](#koop-core), [provider](#provider), [output](output), and [cache](#cache), connected as illustrated below. - -
-### Koop-core +## Koop-core Koop-core is a wrapper around an Express.js server instance. Like Express it listens for incoming requests on a given port and includes some limited middleware for parsing request parameters. It also handles registration of other Koop plugins/components, allowing data and requests to flow through the system. Note that Koop-core ships with a default cache plugin and a default output plugin. As a result -### Provider +## Provider A Koop provider is an interface for connecting with a data source (e.g. Google sheets, Github, local files). It properly formats requests to that data source, and converts the data received from that source to GeoJSON, the common format used by Koop. -### Output +## Output A Koop plugin that transforms the GeoJSON provider by a Koop provider to a specific output format (e.g., GeoServices response, vector-tiles, WMS, etc). It defines service routes and handlers that interpret incoming requests, acquires data from a provider, and post-processes that data before sending it back to the client. Koop-core ships with a default output plugin [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices), so some koop implementation will register no additional output plugins. -### Cache -A Koop cache plugin stores data intially requested from the provider. The cache key is a combination of request parameters. If caching is implemented, Koop will use request parameters to generate a key and check the cache for data. If found and not expired, it is returned to the output plugin without needing to go through the provider and the external data source. If not found, the provider gets the data, but upserts it to the cache with an expiration time. Koop-core ships with a default cache plugin [koop-cache-memory](https://github.com/koopjs/koop-cache-memory), so many some koop implementation will register no cache plugin. \ No newline at end of file +## Cache +A Koop cache plugin stores data intially requested from the provider. The cache key is a combination of request parameters. If caching is implemented, Koop will use request parameters to generate a key and check the cache for data. If found and not expired, it is returned to the output plugin without needing to go through the provider and the external data source. If not found, the provider gets the data, but upserts it to the cache with an expiration time. Koop-core ships with a default cache plugin [koop-cache-memory](https://github.com/koopjs/koop-cache-memory), so many koop implementations will register no cache plugin. diff --git a/_docs/basics/quickstart.md b/_docs/basics/quickstart.md index fea0327..9ada9fa 100644 --- a/_docs/basics/quickstart.md +++ b/_docs/basics/quickstart.md @@ -137,4 +137,3 @@ Once Koop is running, you can test these sample requests:
### Developing with Koop If you want to develop on Koop, it’s usually best to start by creating a Provider. You can read more about that in [provider docs](/documentation/provider) or check out the [Koop Provider Sample](https://github.com/koopjs/koop-provider-sample) - diff --git a/_docs/development/authorization.md b/_docs/development/authorization.md new file mode 100644 index 0000000..6c764fe --- /dev/null +++ b/_docs/development/authorization.md @@ -0,0 +1,61 @@ +--- +title: Authorization Specification +permalink: /docs/development/authorization +--- + +## Authorization plugin specification + +### `index.js` and the registration object + +Each authorization plugin must have a file called `index.js`. `index.js` must be able to deliver a a Koop registration object. This object should have the following minimum content: + +```js +{ + type: 'auth', + authenticationSpecification: Function, + authenticate: Function, + authorize: Function +} +``` + +The object above may be assigned to `module.exports` directly; alternatively, `module.exports` may be assigned an initialization function that returns the registration object on execution. This approach is useful if you need to pass options into the authentication plugin ([see an example](https://github.com/koopjs/koop-auth-direct-file/blob/master/src/index.js#L10-L17)). + +The table below contains a brief description of the registration object. Note that the listed functions make up authorization plugin's API. Koop adds these functions to each provider's `Model` prototype, which makes them available in output plugins. + +| property | type | description | +| --- | --- | --- | +|`type`| String | Must be set to `auth`; identifies the plugin as an authorization plugin during registration | +|`authenticationSpecification`| Function | Returns an object with data useful for configuring the authorization plugin with output plugins.| +|`authorize`| Function | Verfies a user has authorization to make a requests (e.g., a token is validated) | +|`authenticate`|Function| Authenticates a user's requests based on submitted input (credentials, key, etc)| + +Details about each of the API functions are found below. + +#### `authenticationSpecification` **function() ⇒ object** + +Authorization plugins must include a function called "authenticationSpecification". Its purpose is delivery of an object (i.e., the _authentication specification_) that provides options to the output-plugin. The object returned need only contain data for properly configuring your output plugins of choice. For example, Koop's default geoservices uses a `useHttp` option when generating the [authentication endpoint](https://github.com/koopjs/koop-output-geoservices/blob/master/index.js#L54). An example of `authenticationSpecification` is available [here](https://github.com/koopjs/koop-auth-direct-file/blob/master/src/index.js#L44-L56). + +#### `authenticate` **authenticate(req) ⇒ Promise** + +| Param | Type | Description | +| --- | --- | --- | +| req | object | Express request object. Credentials for authentication should be found in the `query` object. | + +Authorization plugins must include a function called `authenticate` that returns a promise. Its purpose is to validate credentials and, if successful, issue a token for authorizing subsequent resource requests. If the authentication is unsuccessful the promise should reject with an error object. The error should have a `code` property with value `401`. If the authentication is successful, the promise should resolve an object with the following properties: + +```js +{ + token: String, // token that can be added to resource requests, and decoded and verified by the "authorization" function + expires: Number, // number seconds until token expires +} +``` + +Authorization plugins are free to validate credentials in any manner. For example, you might check a database for a match of the submitted username and password, or forward the credentials on to a third-party identity-store. [koop-auth-direct-file](https://github.com/koopjs/koop-auth-direct-file) provides an [example](https://github.com/koopjs/koop-auth-direct-file/blob/master/src/index.js#L59-L88) of a very basic credential validation using a simple JSON file-store. + +#### `authorize` function **authorize(req) ⇒ Promise** + +| Param | Type | Description | +| --- | --- | --- | +| req | object | Express request object. Query parameter or header should include input (e.g., token) that can be used to prove previously successful authentication | + +Authorization plugins are required to implement a function called `authorize`. It should accept the Express request object as an argument, which that can be used to verify the request is being made by an authenticated user (e.g., validate a token in the authorization header). If the authorization is unsuccessful, the promise should reject with an error object that contains a `401` code. Successful authorization should allow the promise to resolve. An example of an `authorize` function can be viewed [here](https://github.com/koopjs/koop-auth-direct-file/blob/master/src/index.js#L90-L108). diff --git a/_docs/development/cache.md b/_docs/development/cache.md new file mode 100644 index 0000000..55a0d6e --- /dev/null +++ b/_docs/development/cache.md @@ -0,0 +1,186 @@ +--- +title: Cache Specification +permalink: /docs/development/cache +--- + +Any backend can be used as a Koop cache (e.g. PostGIS, Elasticsearch, Redis etc.). All it must do is follow the specification in this document. + +## API +The cache is a JavaScript object that lives in-memory. It is used to store geojson features. + +### `insert` +Insert geojson into the cache + +Note: A metadata entry will be created automatically. It can be populated from an object on the inserted geojson. + +```js +const geojson = { + type: 'FeatureCollection', + features: [], + metadata: { // Metadata is an arbitrary object that will be stored in the catalog under the same key as the geojson + name: 'Example GeoJSON', + description: 'This is geojson that will be stored in the cache' + } +} + +const options = { + ttl: 1000 // The TTL option is measured in seconds, it will be used to set the `expires` field in the catalog entry +} + +cache.insert('key', geojson, options, err => { + // This function will call back with an error if there is already data in the cache using the same key +}) +``` + +### `append` +Add features to an existing geojson feature collection in the cache +Note: + +```js +const geojson = { + type: 'FeatureCollection', + features: [] +} +cache.append('key', geojson, err => { + // This function will call back with an error if the cache key does not exist +}) +``` + +### `update` +Update a cached feature collection with new features. +This will completely replace the features that are in the cache, but the metadata doc will remain in the catalog. + +```js +const geojson = { + type: 'FeatureCollection', + features: [] +} + +const options = { + ttl: 1000 +} +cache.update('key', geojson, options, err => { + // This function will call back with an error if the cache key does not exist +}) +``` + +### `upsert` +Update a cached feature collection with new features or insert if the features are not there. + +```js +const geojson = { + type: 'FeatureCollection', + features: [] +} + +const options = { + ttl: 1000 +} + +cache.upsert('key', geojson, options, err => { + // This function will call back with an error if the cache key does not exist +}) +``` + +### `retrieve` +Retrieve a cached feature collection, optionally applying a query or aggregation to the data + +```js +const options = {} // This options object may be a query compatible with the GeoServices spec or as is used in Winnow +cache.retrieve('key', options, (err, geojson) => { + /* This function will call back with an error if there is no geojson in the cache + The geojson returned will contain the metadata document from the catalog + { + type: 'FeatureCollection', + features: [], + metadata: {} + } + */ +}) +``` + +### `delete` +Remove a feature collection from the cache + +```js +cache.delete('key', err => { + // This function will call back with an error if there was nothing to delete +}) +``` + +### `createStream` +Create a stream of features from the cache + +```js +cache.createStream('key', options) +.pipe(/* do something with the stream of geojson features emitted one at a time */) +``` + +## Catalog API +The catalog stores metadata about items that are in the cache. + +### `catalog.insert` +Add an arbitrary metadata document to the cache. +Note: This is called automatically by `insert` + +```js +const metadata = { + name: 'Standalone metadata', + status: 'Processing', + description: 'Metadata that is not attached to any other data in the cache' +} + +cache.catalog.insert('key', metadata, err => { + // this function will call back with an error if there is already a metadata document using the same key +}) +``` + +### `catalog.update` +Update a metadata entry + +```js +const original = { + name: 'Doc', + status: 'Processing' +} +cache.catalog.insert('key', original) + +const update = { + status: 'Cached' +} + +cache.catalog.update('key', update, err => { + // this function will call back with an error if there is no metadata in the catalog using that key +}) + +cache.catalog.retrieve('key', (err, metadata) => { + /* + Updates are merged into the existing metadata document + Return value will be: + { + name: 'Doc', + status: 'Cached' + } + */ +}) +``` + +### `catalog.retrieve` +Retrieve a metadata entry from the catalog + +```js +cache.catalog.retrieve('key', (err, metadata) => { + // This function will call back with an error if there is no metadata stored under the given key + // Or else it will call back with the stored metadata doc +}) +``` + +### `catalog.delete` +Remove a catalog entry from the catalog +Note: This cannot be called if data is in the cache under the same key + +```js +cache.catalog.delete('key', err => { + // This function will call back with an error if there is nothing to delete or if there is still data in the cache using the same key +}) +``` \ No newline at end of file diff --git a/_docs/development/output.md b/_docs/development/output.md new file mode 100644 index 0000000..9e9f30f --- /dev/null +++ b/_docs/development/output.md @@ -0,0 +1,8 @@ +--- +title: Output Specification +permalink: /docs/development/output +--- + +## API + +The documentation for an output plugin is not yet ready. See the [koop-output-flat](https://github.com/koopjs/koop-output-flat) source files for an example of a simple output-plugin. \ No newline at end of file diff --git a/_docs/development/provider.md b/_docs/development/provider.md new file mode 100644 index 0000000..b0aa50a --- /dev/null +++ b/_docs/development/provider.md @@ -0,0 +1,278 @@ +--- +title: Provider Specification +permalink: /docs/development/provider +--- + +#### Contents +1. [The `index.js` file](#indexjs) +2. [The `model.js` file](#modeljs) +3. [Routes and Controllers](#routes-and-controllers) + +Note: the discussion of Cached vs Pass-through providers has moved [here](../basics/provider-types). +
+ +## index.js + +Every provider must have a file called `index.js`. Its purpose is to tell Koop how to load and use the provider. The keys and values are enumerated in the example below. + +```js +module.exports = { + name: 'agol', // Required, the name of this provider and the start of all its URLS + type: 'provider', // Required, the type of Koop Plugin this is + version: require('./package.json').version, // Required, the version of this provider + Model: require('./agol'), // Required contains getData and other functions called by controller + hosts: true, // Optional, whether or not routes should include a `host` parameter + disableIdParam: false, // Optional, whether or not routes should include an `id` parameter + routes: require('./routes'), // Optional, any additional routes to be handled by this provider + Controller: require('./controller'), // Optional, a controller to support unique routes +} +``` +
+_[back to top](#contents)_ +
+ +## model.js +Every provider must have a `Model`. This is where almost all of the business logic of the provider will occur. Its primary job is to fetch data from a remote source like an API or database and return GeoJSON to Koop for further processing. `Model` has a set of prototype functions that allow Koop to interact with it. Some of these functions are optional, others are required. The table below lists the `Model` function that Koop currently uses. + +### `Model` class methods overview + +| Name | Required? | Summary | +|----------|:-------------:|------| +| `getData` | Yes | Fetches data and translates it to GeoJSON. See below. | +| `createKey` | No | Generates a string to use as a key for the data-cache. See below. | +| `getAuthenticationSpecification` | No | Delivers an object for use in configuring authentication in output-services. See [authorization spec](http://koopjs.github.io/docs/usage/authorization).| +| `authenticate` | No | Validates credentials and issues a token. See [authorization spec](http://koopjs.github.io/docs/usage/authorization). | +| `authorize` | No | Verifies request is made by an authenticated user. See [authorization spec](http://koopjs.github.io/docs/usage/authorization). | + +_[back to top](#contents)_ + +### `getData` function + +Models are required to implement a function called `getData`. It should fetch data from the remote API, translate the data into GeoJSON (if necessary) and call the `callback` function with the GeoJSON as the second parameter. If there is an error in fetching or processing data from the remote API it should call the `callback` function with an error as the first parameter and stop processing. + +GeoJSON passed to the callback should be valid with respect to the [GeoJSON specification](https://tools.ietf.org/html/rfc7946). Some operations in output-services expect standard GeoJSON properties and / or values. In some cases, having data that conforms to the [GeoJSON spec's righthand rule](https://tools.ietf.org/html/rfc7946#section-3.1.6) is esstential for generating expected results (e.g., features crossing the antimeridian). Koop includes a GeoJSON validation that is suitable for non-production environments and can be activated by setting `NODE_ENV` to anything **except** `production`. In this mode, invalid GeoJSON from `getData` will trigger informative console warnings. + +```js +/* +Example request: http://koop.com/craigslist/washingtondc/apartments/FeatureServer/0/query?where=price>20&f=geojson +Req is the express request object: https://expressjs.com/en/4x/api.html#req + req.params = { + host: 'washingtondc', + id: 'apartments' + } + req.query = { + where: 'price > 20', + f: geojson + } +*/ +function getData(req, callback) { + // pull pieces we need to form a request to Craigslist from the req.params object. + const city = req.params.host + const type = req.params.id + // using the npm package `request` fetch geojson from craigslist + request(`https://${city}.craigslist.org/jsonsearch/type/${type}`, (err, res, body) => { + // if the http request fails stop processing and return and call back with an error + if (err) return callback(err) + // translate the raw response from Craigslist into GeoJSON + const geojson = translate(res.body, type) + // add a little bit of metadata to enrich the geojson + // metadata is handled by the output plugin that is responding to the request. + // e.g. https://github.com/koopjs/koop-output-geoservices + geojson.metadata = { + name: `${city} ${type}`, + description: `Craigslist ${type} listings proxied by https://github.com/dmfenton/koop-provider-craigslist` + } + // hand the geojson back to Koop + callback(null, geojson) + }) +} +``` + +_[back to top](#contents)_ + +#### Setting provider `metadata` in `getData` + +You can add a `metadata` property to the GeoJSON returned from `getData` and assign it an object for use in Koop output services. In addtion to `name` and `description` noted in the example above, the following fields may be useful: + +```js +metadata: { + name: String, // The name of the layer + description: String, // The description of the layer + extent: Array, // valid extent array e.g. [[180,90],[-180,-90]] + displayField: String, // The display field to be used by a client + geometryType: String // REQUIRED if no features are returned with this object Point || MultiPoint || LineString || MultiLineString || Polygon || MultiPolygon + idField: Integer || String, // unique identifier field, + maxRecordCount: Number, // the maximum number of features a provider can return at once + limitExceeded: Boolean, // whether or not the server has limited the features returned + timeInfo: Object // describes the time extent and capabilities of the layer, + transform: Object // describes a quantization transformation + fields: [ + { // Subkeys are optional + name: String, + type: String, // 'Date' || 'Double' || 'Integer' || 'String' + alias: String, // how should clients display this field name + length: Number // max length for a String or Date field (optional) + } + ] + } +``` + +The data type and values used for `idField` can affect the output of the [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices) and behavior of some consumer clients. [FeatureServer](https://github.com/koopjs/FeatureServer) and [winnow](https://github.com/koopjs/winnow) (dependencies of [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices)) will create a separate OBJECTID field and set its value to the value of the attribute referenced by `idField`. OBJECTIDs that are not integers or outside the range of 0 - 2,147,483,647 can break features in some ArcGIS clients. + +_[back to top](#contents)_ + +#### Request parameters in `getData` + +Recall the `getData` function receives `req`, an Express.js [request](https://expressjs.com/en/4x/api.html#req) object. `req` includes a set of [route parameters](https://expressjs.com/en/4x/api.html#req.params) accessible with `req.params`, as well as a set of [query-parameters](https://expressjs.com/en/4x/api.html#req.query) accessible with `req.query`. Parameters can be used by `getData` to specify the particulars of data fetching. + +##### Provider route parameters +Providers can enable the `:host` and `:id` route parameters with settings in [`index.js`](#index.js) and then leverage these parameters in `getData`. For example, the Craigslist Provider's [`getData` function](https://github.com/dmfenton/koop-provider-craigslist/blob/master/model.js#L12-L14) uses the `:host` and `:id` parameter to transmit the city and and posting category to the `getData` function where they are used to generate URLs for requests to the Craigslist API. + +| parameter | `getData` accessor | enabled by default | `index.js` setting | +| --- | --- | --- | --- | +| `:id` | `req.params.id` | yes | `disableIdParam` | +|`:host`| `req.params.host` | no | `hosts` | + +##### Output-services route parameters + +By default, Koop includes the [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices) output-service. It adds a set of `FeatureServer` routes (e.g., `/provider/:host/:id/FeatureServer/:layer` and `/provider/:host/:id/FeatureServer/:layer/:method`), which include addtional route parameters that can be used in your Model's `getData` function. + +| parameter | `getData` accessor | notes| +| --- | --- | --- | +| `:layer` | `req.params.layer` | | +|`:method`| `req.params.method` | `query` and `generateRenderer` are the only values currently supported by [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices) dependency. All other values will produce a `400, Method not supported` error. | + +##### Query parameters +As noted above, any query-parameters added to the request URL can accessed within `getData` and leveraged for data fetching purposes. For example, a request `/provider/:id/FeatureServer/0?foo=bar` would supply `getData` with `req.query.foo` equal to `bar`. With the proper logic, it could then be used to limit fetched data to records that had an attribute `foo` with a value of `bar`. + +#### Generation of provider-specific output-routes +The position of the provider-specific fragment of a route path can vary depending on the `path` assignment in the `routes` array object of your output-services plugin. By default, Koop will construct the route with the provider's parameters first, and subsequently add the route fragment defined by an output-services plugin. However, if you need the route path configured differently, you can add the `$namespace` and `$providerParams` placholders anywhere in the output-services path. Koop will replace these placeholders with the provider-specific route fragments (i.e, namespace and `:host/:id`). For example, an output path defined as `$namespace/rest/services/$providerParams/FeatureServer/0` would translate to `provider/rest/services/:host/:id/FeatureServer/0`. + +#### Output-routes without provider parameters +You may need routes that skip the addition of provider-specific parameters altogether. This can be accomplished by adding an `absolutePath: true` key-value to the `routes` array object in your output-services plugin. On such routes, Koop will define the route without any additional provider namespace or parameters. + +_[back to top](#contents)_ + +### `createKey` function +Koop uses a an internal `createKey` function to generate a string for use as a key for the data-cache's key-value store. Koop's `createKey` uses the provider name and route parameters to define a key. This allows all requests with the same provider name and route parameters to leverage cached data. + +Models can optionally implement a function called `createKey`. If defined, the Model's `createKey` overrides Koop's internal function. This can be useful if the cache key should be composed with parameters in addition to those found in the internal function. For example, the `createKey` below uses query parameters `startdate` and `enddate` to construct the key (if they are defined): + +```js +Model.prototype.createKey = function (req) { + let key = req.url.split('/')[1] + if (req.params.host) key = `${key}::${req.params.host}` + if (req.params.id) key = `${key}::${req.params.id}` + key = (req.query.startdate && req.query.enddate) ? `${key}::${req.params.startdate}::${req.params.enddate}` : '' + return key +} +``` +_[back to top](#contents)_ + +## Routes and Controllers + +### Routes.js + +This file is simply an array of routes that should be handled in the namespace of the provider e.g. http://adapters.koopernetes.com/agol/arcgis/datasets/e5255b1f69944bcd9cf701025b68f411_0 + +In the example above, the provider namespace is `agol`, and `arcgis` and `e5255b1f69944bcd9cf701025b68f411_0` are parameters. Anything that matches `/agol/*/datasets/*` will be be handled by the model. + +Each route has: +- path: an express.js style route that includes optional parameters + - see https://www.npmjs.com/package/path-to-regexp for details +- methods: the http methods that should be handled at this route + - e.g. `get`, `post`, `put`, `delete`, `patch` +- handler: the Controller function that should handle requests at this route + +Example: + +```js +module.exports = [ +{ + path: '/agol/:id/datasets', + methods: ['get'], + handler: 'get' + }, + { + path: '/agol/:id/datasets/:dataset.:format', + methods: ['get'], + handler: 'get' + }, + { + path: '/agol/:id/datasets/:dataset', + methods: ['get'], + handler: 'get' + }, + { + path: '/agol/:id/datasets/:dataset/:method', + methods: ['delete'], + handler: 'post' + }, + { + path: '/agol/:id/datasets/:dataset/:method', + methods: ['delete'], + handler: 'delete' + } +] +``` + +### Controller.js + +Providers can do more than simply implement `getData` and hand GeoJSON back to Koop's core. In fact, they can extend the API space in an arbitrary fashion by adding routes that map to controller functions. Those controller functions call functions on the model to fetch or process data from the remote API. + +The purpose of the Controller file is to handle additional routes that are specified in `route.js`. It is a class that is instantiated with access to the model. Most processing should happen in the model, while the controller acts as a traffic director. + +As of Koop 3.0, you **do not** need to create a controller. If you want to add additional functionality to Koop that is not supported by an output plugin, you can add additional functions to the controller. + +Example: + +```js +module.exports = function (model) { + this.model = model + /** + * returns a list of the registered hosts and their ids + */ + this.get = function (req, res) { + this.model.log.debug({route: 'dataset:get', params: req.params, query: req.query}) + if (req.params.dataset) this._getDataset(req, res) + else this._getDatasets(req, res) + } + + /** + * Put a specific dataset on the queue + */ + this.post = function (req, res) { + this.model.log.debug({route: 'POST', params: req.params, query: req.query}) + if (req.params.method === 'import') this.model.enqueue('import', req, res) + else if (req.params.method === 'export') this.model.enqueue('export', req, res) + else return res.status(400).json({error: 'Unsupported method'}) + } + + /** + * Deletes a dataset from the cache + */ + this.delete = function (req, res) { + this.model.log.debug(JSON.stringify({route: 'dataset:delete', params: req.params, query: req.query})) + var ids = this.model.decomposeId(req.params.dataset) + this.model.dropResource(ids.item, ids.layer, {}, function (err) { + if (err) return res.status(500).send({error: err.message}) + res.status(200).json({status: 'Deleted'}) + }) + } + + this._getDataset = function (req, res) { + this.findRecord(req.params, function (err, dataset) { + if (err) return res.status(404).json({error: err.message}) + res.status(200).json({dataset: dataset}) + }) + } + + this._getDatasets = function (req, res) { + this.model.findRecords(req.query, function (err, datasets) { + if (err) return res.status(500).json({error: err.message}) + res.status(200).json({datasets: datasets}) + }) + } +} +``` +_[back to top](#contents)_ diff --git a/_docs/usage/authorization.md b/_docs/usage/authorization.md index 3e77c32..0d5a283 100644 --- a/_docs/usage/authorization.md +++ b/_docs/usage/authorization.md @@ -1,5 +1,5 @@ --- -title: Authorization Specification +title: Authorization permalink: /docs/usage/authorization --- @@ -13,9 +13,7 @@ Authorization plugins add a security layer to Koop. There are few important fact * Authoriztion plugins should be registered _after_ output-services plugins and _before_ providers. Any providers registered before an authorization plugin will not have its plugin routes secured. -Dive into the docs below or check out an existing auth [plugin](https://github.com/koopjs/koop-auth-direct-file) as well as its interface in Koop's default [geoservices output plugin](https://github.com/koopjs/koop-output-geoservices/blob/master/index.js). - -### Usage +## Usage Usage of an authorization-plugin can be conceptually divided into three parts: (1) configuration, (2) registration with Koop, and (3) use in output-services. An example using [koop-auth-direct-file](https://github.com/koopjs/koop-auth-direct-file) provides the best illustration of usage. @@ -42,61 +40,9 @@ const s3Select = require('@koopjs/provider-s3-select') koop.register(s3Select) ``` -In the above implementation, the authorization plugin would be applied to feature server routes for the S3 Select provider. Note that the routes for the Github provider _would not_ be protected because the auth plugin registration occurs _after_ the Github provider registration. - -## Authorization plugin specification - -### `index.js` and the registration object - -Each authorization plugin must have a file called `index.js`. `index.js` must be able to deliver a a Koop registration object. This object should have the following minimum content: - -```js -{ - type: 'auth', - authenticationSpecification: Function, - authenticate: Function, - authorize: Function -} -``` - -The object above may be assigned to `module.exports` directly; alternatively, `module.exports` may be assigned an initialization function that returns the registration object on execution. This approach is useful if you need to pass options into the authentication plugin ([see an example](https://github.com/koopjs/koop-auth-direct-file/blob/master/src/index.js#L10-L17)). - -The table below contains a brief description of the registration object. Note that the listed functions make up authorization plugin's API. Koop adds these functions to each provider's `Model` prototype, which makes them available in output plugins. - -| property | type | description | -| --- | --- | --- | -|`type`| String | Must be set to `auth`; identifies the plugin as an authorization plugin during registration | -|`authenticationSpecification`| Function | Returns an object with data useful for configuring the authorization plugin with output plugins.| -|`authorize`| Function | Verfies a user has authorization to make a requests (e.g., a token is validated) | -|`authenticate`|Function| Authenticates a user's requests based on submitted input (credentials, key, etc)| - -Details about each of the API functions are found below. - -#### `authenticationSpecification` **function() ⇒ object** - -Authorization plugins must include a function called "authenticationSpecification". Its purpose is delivery of an object (i.e., the _authentication specification_) that provides options to the output-plugin. The object returned need only contain data for properly configuring your output plugins of choice. For example, Koop's default geoservices uses a `useHttp` option when generating the [authentication endpoint](https://github.com/koopjs/koop-output-geoservices/blob/master/index.js#L54). An example of `authenticationSpecification` is available [here](https://github.com/koopjs/koop-auth-direct-file/blob/master/src/index.js#L44-L56). - -#### `authenticate` **authenticate(req) ⇒ Promise** - -| Param | Type | Description | -| --- | --- | --- | -| req | object | Express request object. Credentials for authentication should be found in the `query` object. | - -Authorization plugins must include a function called `authenticate` that returns a promise. Its purpose is to validate credentials and, if successful, issue a token for authorizing subsequent resource requests. If the authentication is unsuccessful the promise should reject with an error object. The error should have a `code` property with value `401`. If the authentication is successful, the promise should resolve an object with the following properties: - -```js -{ - token: String, // token that can be added to resource requests, and decoded and verified by the "authorization" function - expires: Number, // number seconds until token expires -} -``` - -Authorization plugins are free to validate credentials in any manner. For example, you might check a database for a match of the submitted username and password, or forward the credentials on to a third-party identity-store. [koop-auth-direct-file](https://github.com/koopjs/koop-auth-direct-file) provides an [example](https://github.com/koopjs/koop-auth-direct-file/blob/master/src/index.js#L59-L88) of a very basic credential validation using a simple JSON file-store. - -#### `authorize` function **authorize(req) ⇒ Promise** - -| Param | Type | Description | -| --- | --- | --- | -| req | object | Express request object. Query parameter or header should include input (e.g., token) that can be used to prove previously successful authentication | +## Plugin registration order +The order in which you register your providers and authorization plugins will affect functionality. The key points are: +* Providers registered *before* the authorization plugin *will not* be secured +* Providers registered *after* the authorization plugin *will* be secured -Authorization plugins are required to implement a function called `authorize`. It should accept the Express request object as an argument, which that can be used to verify the request is being made by an authenticated user (e.g., validate a token in the authorization header). If the authorization is unsuccessful, the promise should reject with an error object that contains a `401` code. Successful authorization should allow the promise to resolve. An example of an `authorize` function can be viewed [here](https://github.com/koopjs/koop-auth-direct-file/blob/master/src/index.js#L90-L108). +In the above implementation, the authorization plugin would be applied to feature server routes for the S3 Select provider. Note that the routes for the Github provider _would not_ be protected because the auth plugin registration occurs _after_ the Github provider registration. \ No newline at end of file diff --git a/_docs/usage/cache.md b/_docs/usage/cache.md index 1a45df9..eea8a88 100644 --- a/_docs/usage/cache.md +++ b/_docs/usage/cache.md @@ -1,9 +1,11 @@ --- -title: Cache Specification +title: Cache permalink: /docs/usage/cache --- -Any backend can be used as a Koop cache (e.g. PostGIS, Elasticsearch, Redis etc.). All it must do is follow the specification in this document. +## Overview + +A Koop cache plugin stores data requested from the provider for use in followup requests. The cache key is a combination of request parameters. If caching is implemented, Koop will use request parameters to generate a key and check the cache for data. If found and not expired, it is returned to the output plugin without needing to go through the provider and the external data source. If not found, the provider gets the data, but upserts it to the cache with an expiration time. Koop-core ships with a default cache plugin [koop-cache-memory](https://github.com/koopjs/koop-cache-memory), so many koop implementations will register no cache plugin. ## Usage @@ -12,187 +14,7 @@ To be used by Koop, the cache must be registered with the Koop object before the ```js const Koop = require('koop') const koop = new Koop() -const cache = require('koop-cache-memory') +const cache = require('koop-cache-redis') koop.register(cache) koop.server.listen(80) ``` - -## API -The cache is a JavaScript object that lives in-memory. It is used to store geojson features. - -### `insert` -Insert geojson into the cache - -Note: A metadata entry will be created automatically. It can be populated from an object on the inserted geojson. - -```js -const geojson = { - type: 'FeatureCollection', - features: [], - metadata: { // Metadata is an arbitrary object that will be stored in the catalog under the same key as the geojson - name: 'Example GeoJSON', - description: 'This is geojson that will be stored in the cache' - } -} - -const options = { - ttl: 1000 // The TTL option is measured in seconds, it will be used to set the `expires` field in the catalog entry -} - -cache.insert('key', geojson, options, err => { - // This function will call back with an error if there is already data in the cache using the same key -}) -``` - -### `append` -Add features to an existing geojson feature collection in the cache -Note: - -```js -const geojson = { - type: 'FeatureCollection', - features: [] -} -cache.append('key', geojson, err => { - // This function will call back with an error if the cache key does not exist -}) -``` - -### `update` -Update a cached feature collection with new features. -This will completely replace the features that are in the cache, but the metadata doc will remain in the catalog. - -```js -const geojson = { - type: 'FeatureCollection', - features: [] -} - -const options = { - ttl: 1000 -} -cache.update('key', geojson, options, err => { - // This function will call back with an error if the cache key does not exist -}) -``` - -### `upsert` -Update a cached feature collection with new features or insert if the features are not there. - -```js -const geojson = { - type: 'FeatureCollection', - features: [] -} - -const options = { - ttl: 1000 -} - -cache.upsert('key', geojson, options, err => { - // This function will call back with an error if the cache key does not exist -}) -``` - -### `retrieve` -Retrieve a cached feature collection, optionally applying a query or aggregation to the data - -```js -const options = {} // This options object may be a query compatible with the GeoServices spec or as is used in Winnow -cache.retrieve('key', options, (err, geojson) => { - /* This function will call back with an error if there is no geojson in the cache - The geojson returned will contain the metadata document from the catalog - { - type: 'FeatureCollection', - features: [], - metadata: {} - } - */ -}) -``` - -### `delete` -Remove a feature collection from the cache - -```js -cache.delete('key', err => { - // This function will call back with an error if there was nothing to delete -}) -``` - -### `createStream` -Create a stream of features from the cache - -```js -cache.createStream('key', options) -.pipe(/* do something with the stream of geojson features emitted one at a time */) -``` - -## Catalog API -The catalog stores metadata about items that are in the cache. - -### `catalog.insert` -Add an arbitrary metadata document to the cache. -Note: This is called automatically by `insert` - -```js -const metadata = { - name: 'Standalone metadata', - status: 'Processing', - description: 'Metadata that is not attached to any other data in the cache' -} - -cache.catalog.insert('key', metadata, err => { - // this function will call back with an error if there is already a metadata document using the same key -}) -``` - -### `catalog.update` -Update a metadata entry - -```js -const original = { - name: 'Doc', - status: 'Processing' -} -cache.catalog.insert('key', original) - -const update = { - status: 'Cached' -} - -cache.catalog.update('key', update, err => { - // this function will call back with an error if there is no metadata in the catalog using that key -}) - -cache.catalog.retrieve('key', (err, metadata) => { - /* - Updates are merged into the existing metadata document - Return value will be: - { - name: 'Doc', - status: 'Cached' - } - */ -}) -``` - -### `catalog.retrieve` -Retrieve a metadata entry from the catalog - -```js -cache.catalog.retrieve('key', (err, metadata) => { - // This function will call back with an error if there is no metadata stored under the given key - // Or else it will call back with the stored metadata doc -}) -``` - -### `catalog.delete` -Remove a catalog entry from the catalog -Note: This cannot be called if data is in the cache under the same key - -```js -cache.catalog.delete('key', err => { - // This function will call back with an error if there is nothing to delete or if there is still data in the cache using the same key -}) -``` \ No newline at end of file diff --git a/_docs/usage/koop-core.md b/_docs/usage/koop-core.md index 961c9ca..1e72340 100644 --- a/_docs/usage/koop-core.md +++ b/_docs/usage/koop-core.md @@ -6,7 +6,7 @@ redirect_from: /docs/usage/index.html Koop-core and its implementation in the main `server.js` file is where all providers, output-services, caches, and authorization plugins are registered. It exposes an [Express](https://expressjs.com) server which can be instructed to listen on port 80 or any port of your choosing. This doc walks through the steps for creating this file from scratch. -## Create the package.json +## package.json Since you need the Koop `npm` dependency, you should start by creating a `package.json` file. With Node.js installed, run: ```bash @@ -20,9 +20,8 @@ npm install --save koop npm install --save @koopjs/provider-github ```
-
-## Create the server.js +## server.js Create a new file named `server.js`. The typical content of the Koop server file look like the snippet below: @@ -50,10 +49,25 @@ koop.server.listen(8080, () => console.log(`Koop listening on port 8080!`)) ```
-
+ +## Configuration settings + +Koop leverages the [config](https://www.npmjs.com/package/config) package to configuration settings. To faciliate this module, you should add a `config` directory to your Koop instance with a `default.json` file. See the [config](https://www.npmjs.com/package/config) documentation for details on using different JSON files and environment variables to vary configuration. + +Many Koop plugins will require settings in your config file. See each plugin's documentation for specifics. Configuration settings for the Koop server are noted below. + +### disableCompression +As of v3.12.0, Koop adds Express compression by default. If you do not want Express compression (e.g., perhaps you are using Nginx for compression), you can disable it by adding a `disableCompression` boolean to your Koop config file: + +```json +{ + "disableCompression": true +} +``` + ## Running a Koop instance -Run the `server.js` file like an Node.js script. +Run the `server.js` file like any Node.js script. ```bash > node server.js @@ -104,53 +118,17 @@ Koop listening on port 8080! ``` ### Interpreting the console output -You will notice a list of routes printed to the console. These include routes for the built-in `datasets` provider, as well as any providers you have registered. Each provider will have a list of any custom routes and a list of routes defined by registered output plugins. You can use the listed routes by appending them to the `host:port` of your running Koop instance and replace any parameters with values. For example: +You will notice a list of routes printed to the console. These include routes for the built-in `datasets` provider, as well as any providers you have registered (here, the Github provider). Each provider will have a list of any custom routes and a list of routes defined by registered output plugins. You can use the listed routes by appending them to the `host:port` of your running Koop instance and replace any parameters with values. For example: `/github/rest/services/:id/FeatureServer/:layer/:method` becomes: -`http://localhost:3000/github/koopjs::geodata::north-america/FeatureServer/0/query` +`http://localhost:8080/github/koopjs::geodata::north-america/FeatureServer/0/query` Note that there are `GET` and `POST` versions of all koop-output-geoservices routes. Output-services define an array of methods for each of their routes, and in this case every route has been [assigned both `GET` and `POST` methods](https://github.com/koopjs/koop-output-geoservices/blob/master/index.js#L94).
-
- -## Koop options - -### Disable compression -As of v3.12.0, Koop adds Express compression by default. If you do not want Express compression, you can disable it by adding a `disableCompression: true` to your Koop config file. - -### Route prefixing -If needed, you can add a prefix to all of a registered provider's routes. For example, if you wanted the fragment `/api/v1` prepended to you github provider routes you could register the provider like this: - -```js -const provider = require('@koopjs/provider-github') -koop.register(provider, { routePrefix: '/api/v1'}) -``` - -which results in routes like: - -```bash -Routes for provider: github, Output: Geoservices Methods -------------------------------------------------------------- --------- -/api/v1/github/rest/info GET, POST -/api/v1/github/tokens/:method GET, POST -/api/v1/github/tokens/ GET, POST -/api/v1/github/rest/services/:id/FeatureServer/:layer/:method GET, POST -/api/v1/github/rest/services/:id/FeatureServer/layers GET, POST -/api/v1/github/rest/services/:id/FeatureServer/:layer GET, POST -/api/v1/github/rest/services/:id/FeatureServer GET, POST -/api/v1/github/:id/FeatureServer/:layer/:method GET, POST -/api/v1/github/:id/FeatureServer/layers GET, POST -/api/v1/github/:id/FeatureServer/:layer GET, POST -/api/v1/github/:id/FeatureServer GET, POST -/api/v1/github/rest/services/:id/FeatureServer* GET, POST -/api/v1/github/:id/FeatureServer* GET, POST -/api/v1/github/rest/services/:id/MapServer* GET, POST -/api/v1/github/:id/MapServer* GET, POST -``` ## Koop as middleware diff --git a/_docs/usage/output.md b/_docs/usage/output.md index d6061c4..b479819 100644 --- a/_docs/usage/output.md +++ b/_docs/usage/output.md @@ -5,6 +5,8 @@ permalink: /docs/usage/output ## Usage +By default, Koop registers the [Geoservices output-plugin](https://github.com/koopjs/koop-output-geoservices) by default. If you want to use additional output-plugins you will have to register them with you Koop instance. In your Koop server file: + To be used by Koop, the output-plugin must be registered with the Koop instance before the it begins listening. ```js @@ -15,6 +17,19 @@ koop.register(output) koop.server.listen(80) ``` -## API +## Plugin registration order +The order in which you register plugins can affect functionality. The key points are: + +* An output-plugin must be registered with the Koop instance before the it begins listening. +* To secure output-plugin routes, it must be registered before an authorization plugin +* To pair and output-plugin with a provider, it must be registered before the provider + +### Using Koop CLI + +If you are using Koop CLI, you can add an output plugin with the `add` command: + +```bash +koop add output +``` -The documentation for an output plugin is not yet ready. See the [koop-output-flat](https://github.com/koopjs/koop-output-flat) source files for an example of a simple output-plugin. \ No newline at end of file +See the Koop CLI [documentation](https://github.com/koopjs/koop-cli#add) for additional options and details. \ No newline at end of file diff --git a/_docs/usage/provider.md b/_docs/usage/provider.md index 8179673..8a676e8 100644 --- a/_docs/usage/provider.md +++ b/_docs/usage/provider.md @@ -1,278 +1,43 @@ --- -title: Provider Specification +title: Provider Usage permalink: /docs/usage/provider --- -#### Contents -1. [The `index.js` file](#indexjs) -2. [The `model.js` file](#modeljs) -3. [Routes and Controllers](#routes-and-controllers) - -Note: the discussion of Cached vs Pass-through providers has moved [here](../basics/provider-types). -
- -## index.js - -Every provider must have a file called `index.js`. Its purpose is to tell Koop how to load and use the provider. The keys and values are enumerated in the example below. +In your Koop server file, the provider must be registered with the Koop instance before the it begins listening. ```js -module.exports = { - name: 'agol', // Required, the name of this provider and the start of all its URLS - type: 'provider', // Required, the type of Koop Plugin this is - version: require('./package.json').version, // Required, the version of this provider - Model: require('./agol'), // Required contains getData and other functions called by controller - hosts: true, // Optional, whether or not routes should include a `host` parameter - disableIdParam: false, // Optional, whether or not routes should include an `id` parameter - routes: require('./routes'), // Optional, any additional routes to be handled by this provider - Controller: require('./controller'), // Optional, a controller to support unique routes -} +const Koop = require('koop') +const koop = new Koop() +const provider = require('') +koop.register(output) +koop.server.listen(80) ``` -
-_[back to top](#contents)_ -
- -## model.js -Every provider must have a `Model`. This is where almost all of the business logic of the provider will occur. Its primary job is to fetch data from a remote source like an API or database and return GeoJSON to Koop for further processing. `Model` has a set of prototype functions that allow Koop to interact with it. Some of these functions are optional, others are required. The table below lists the `Model` function that Koop currently uses. - -### `Model` class methods overview - -| Name | Required? | Summary | -|----------|:-------------:|------| -| `getData` | Yes | Fetches data and translates it to GeoJSON. See below. | -| `createKey` | No | Generates a string to use as a key for the data-cache. See below. | -| `getAuthenticationSpecification` | No | Delivers an object for use in configuring authentication in output-services. See [authorization spec](http://koopjs.github.io/docs/usage/authorization).| -| `authenticate` | No | Validates credentials and issues a token. See [authorization spec](http://koopjs.github.io/docs/usage/authorization). | -| `authorize` | No | Verifies request is made by an authenticated user. See [authorization spec](http://koopjs.github.io/docs/usage/authorization). | -_[back to top](#contents)_ +## Plugin registration order +The order in which you register plugins can affect functionality. The key points are: +* Provider data will only be accessible from output-plugin routes if the provider is registered _after_ the output-plugin +* Provider data will only be secured by an authorization plugin if the provider is registered _after_ the authorization-plugin. -### `getData` function +You can therefore micro-manage provider accessibility by adjusting the registration order of various plugins. -Models are required to implement a function called `getData`. It should fetch data from the remote API, translate the data into GeoJSON (if necessary) and call the `callback` function with the GeoJSON as the second parameter. If there is an error in fetching or processing data from the remote API it should call the `callback` function with an error as the first parameter and stop processing. - -GeoJSON passed to the callback should be valid with respect to the [GeoJSON specification](https://tools.ietf.org/html/rfc7946). Some operations in output-services expect standard GeoJSON properties and / or values. In some cases, having data that conforms to the [GeoJSON spec's righthand rule](https://tools.ietf.org/html/rfc7946#section-3.1.6) is esstential for generating expected results (e.g., features crossing the antimeridian). Koop includes a GeoJSON validation that is suitable for non-production environments and can be activated by setting `NODE_ENV` to anything **except** `production`. In this mode, invalid GeoJSON from `getData` will trigger informative console warnings. +## Route prefixing +If needed, you can add a prefix to all of a registered provider's routes. For example, if you wanted the fragment `/api/v1` prepended to you github provider routes you could register the provider like this: ```js -/* -Example request: http://koop.com/craigslist/washingtondc/apartments/FeatureServer/0/query?where=price>20&f=geojson -Req is the express request object: https://expressjs.com/en/4x/api.html#req - req.params = { - host: 'washingtondc', - id: 'apartments' - } - req.query = { - where: 'price > 20', - f: geojson - } -*/ -function getData(req, callback) { - // pull pieces we need to form a request to Craigslist from the req.params object. - const city = req.params.host - const type = req.params.id - // using the npm package `request` fetch geojson from craigslist - request(`https://${city}.craigslist.org/jsonsearch/type/${type}`, (err, res, body) => { - // if the http request fails stop processing and return and call back with an error - if (err) return callback(err) - // translate the raw response from Craigslist into GeoJSON - const geojson = translate(res.body, type) - // add a little bit of metadata to enrich the geojson - // metadata is handled by the output plugin that is responding to the request. - // e.g. https://github.com/koopjs/koop-output-geoservices - geojson.metadata = { - name: `${city} ${type}`, - description: `Craigslist ${type} listings proxied by https://github.com/dmfenton/koop-provider-craigslist` - } - // hand the geojson back to Koop - callback(null, geojson) - }) -} +const provider = require('@koopjs/provider-github') +koop.register(provider, { routePrefix: '/api/v1'}) ``` -_[back to top](#contents)_ - -#### Setting provider `metadata` in `getData` - -You can add a `metadata` property to the GeoJSON returned from `getData` and assign it an object for use in Koop output services. In addtion to `name` and `description` noted in the example above, the following fields may be useful: - -```js -metadata: { - name: String, // The name of the layer - description: String, // The description of the layer - extent: Array, // valid extent array e.g. [[180,90],[-180,-90]] - displayField: String, // The display field to be used by a client - geometryType: String // REQUIRED if no features are returned with this object Point || MultiPoint || LineString || MultiLineString || Polygon || MultiPolygon - idField: Integer || String, // unique identifier field, - maxRecordCount: Number, // the maximum number of features a provider can return at once - limitExceeded: Boolean, // whether or not the server has limited the features returned - timeInfo: Object // describes the time extent and capabilities of the layer, - transform: Object // describes a quantization transformation - fields: [ - { // Subkeys are optional - name: String, - type: String, // 'Date' || 'Double' || 'Integer' || 'String' - alias: String, // how should clients display this field name - length: Number // max length for a String or Date field (optional) - } - ] - } -``` - -The data type and values used for `idField` can affect the output of the [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices) and behavior of some consumer clients. [FeatureServer](https://github.com/koopjs/FeatureServer) and [winnow](https://github.com/koopjs/winnow) (dependencies of [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices)) will create a separate OBJECTID field and set its value to the value of the attribute referenced by `idField`. OBJECTIDs that are not integers or outside the range of 0 - 2,147,483,647 can break features in some ArcGIS clients. - -_[back to top](#contents)_ - -#### Request parameters in `getData` - -Recall the `getData` function receives `req`, an Express.js [request](https://expressjs.com/en/4x/api.html#req) object. `req` includes a set of [route parameters](https://expressjs.com/en/4x/api.html#req.params) accessible with `req.params`, as well as a set of [query-parameters](https://expressjs.com/en/4x/api.html#req.query) accessible with `req.query`. Parameters can be used by `getData` to specify the particulars of data fetching. - -##### Provider route parameters -Providers can enable the `:host` and `:id` route parameters with settings in [`index.js`](#index.js) and then leverage these parameters in `getData`. For example, the Craigslist Provider's [`getData` function](https://github.com/dmfenton/koop-provider-craigslist/blob/master/model.js#L12-L14) uses the `:host` and `:id` parameter to transmit the city and and posting category to the `getData` function where they are used to generate URLs for requests to the Craigslist API. - -| parameter | `getData` accessor | enabled by default | `index.js` setting | -| --- | --- | --- | --- | -| `:id` | `req.params.id` | yes | `disableIdParam` | -|`:host`| `req.params.host` | no | `hosts` | - -##### Output-services route parameters - -By default, Koop includes the [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices) output-service. It adds a set of `FeatureServer` routes (e.g., `/provider/:host/:id/FeatureServer/:layer` and `/provider/:host/:id/FeatureServer/:layer/:method`), which include addtional route parameters that can be used in your Model's `getData` function. - -| parameter | `getData` accessor | notes| -| --- | --- | --- | -| `:layer` | `req.params.layer` | | -|`:method`| `req.params.method` | `query` and `generateRenderer` are the only values currently supported by [koop-output-geoservices](https://github.com/koopjs/koop-output-geoservices) dependency. All other values will produce a `400, Method not supported` error. | - -##### Query parameters -As noted above, any query-parameters added to the request URL can accessed within `getData` and leveraged for data fetching purposes. For example, a request `/provider/:id/FeatureServer/0?foo=bar` would supply `getData` with `req.query.foo` equal to `bar`. With the proper logic, it could then be used to limit fetched data to records that had an attribute `foo` with a value of `bar`. - -#### Generation of provider-specific output-routes -The position of the provider-specific fragment of a route path can vary depending on the `path` assignment in the `routes` array object of your output-services plugin. By default, Koop will construct the route with the provider's parameters first, and subsequently add the route fragment defined by an output-services plugin. However, if you need the route path configured differently, you can add the `$namespace` and `$providerParams` placholders anywhere in the output-services path. Koop will replace these placeholders with the provider-specific route fragments (i.e, namespace and `:host/:id`). For example, an output path defined as `$namespace/rest/services/$providerParams/FeatureServer/0` would translate to `provider/rest/services/:host/:id/FeatureServer/0`. +which results in routes like: -#### Output-routes without provider parameters -You may need routes that skip the addition of provider-specific parameters altogether. This can be accomplished by adding an `absolutePath: true` key-value to the `routes` array object in your output-services plugin. On such routes, Koop will define the route without any additional provider namespace or parameters. +`/api/v1/github/:id/FeatureServer` -_[back to top](#contents)_ +## Koop CLI -### `createKey` function -Koop uses a an internal `createKey` function to generate a string for use as a key for the data-cache's key-value store. Koop's `createKey` uses the provider name and route parameters to define a key. This allows all requests with the same provider name and route parameters to leverage cached data. +If you are using the Koop CLI, adding a new provider to you Koop application is easy: -Models can optionally implement a function called `createKey`. If defined, the Model's `createKey` overrides Koop's internal function. This can be useful if the cache key should be composed with parameters in addition to those found in the internal function. For example, the `createKey` below uses query parameters `startdate` and `enddate` to construct the key (if they are defined): - -```js -Model.prototype.createKey = function (req) { - let key = req.url.split('/')[1] - if (req.params.host) key = `${key}::${req.params.host}` - if (req.params.id) key = `${key}::${req.params.id}` - key = (req.query.startdate && req.query.enddate) ? `${key}::${req.params.startdate}::${req.params.enddate}` : '' - return key -} -``` -_[back to top](#contents)_ - -## Routes and Controllers - -### Routes.js - -This file is simply an array of routes that should be handled in the namespace of the provider e.g. http://adapters.koopernetes.com/agol/arcgis/datasets/e5255b1f69944bcd9cf701025b68f411_0 - -In the example above, the provider namespace is `agol`, and `arcgis` and `e5255b1f69944bcd9cf701025b68f411_0` are parameters. Anything that matches `/agol/*/datasets/*` will be be handled by the model. - -Each route has: -- path: an express.js style route that includes optional parameters - - see https://www.npmjs.com/package/path-to-regexp for details -- methods: the http methods that should be handled at this route - - e.g. `get`, `post`, `put`, `delete`, `patch` -- handler: the Controller function that should handle requests at this route - -Example: - -```js -module.exports = [ -{ - path: '/agol/:id/datasets', - methods: ['get'], - handler: 'get' - }, - { - path: '/agol/:id/datasets/:dataset.:format', - methods: ['get'], - handler: 'get' - }, - { - path: '/agol/:id/datasets/:dataset', - methods: ['get'], - handler: 'get' - }, - { - path: '/agol/:id/datasets/:dataset/:method', - methods: ['delete'], - handler: 'post' - }, - { - path: '/agol/:id/datasets/:dataset/:method', - methods: ['delete'], - handler: 'delete' - } -] +```bash +> koop add provider ``` -### Controller.js - -Providers can do more than simply implement `getData` and hand GeoJSON back to Koop's core. In fact, they can extend the API space in an arbitrary fashion by adding routes that map to controller functions. Those controller functions call functions on the model to fetch or process data from the remote API. - -The purpose of the Controller file is to handle additional routes that are specified in `route.js`. It is a class that is instantiated with access to the model. Most processing should happen in the model, while the controller acts as a traffic director. - -As of Koop 3.0, you **do not** need to create a controller. If you want to add additional functionality to Koop that is not supported by an output plugin, you can add additional functions to the controller. - -Example: - -```js -module.exports = function (model) { - this.model = model - /** - * returns a list of the registered hosts and their ids - */ - this.get = function (req, res) { - this.model.log.debug({route: 'dataset:get', params: req.params, query: req.query}) - if (req.params.dataset) this._getDataset(req, res) - else this._getDatasets(req, res) - } - - /** - * Put a specific dataset on the queue - */ - this.post = function (req, res) { - this.model.log.debug({route: 'POST', params: req.params, query: req.query}) - if (req.params.method === 'import') this.model.enqueue('import', req, res) - else if (req.params.method === 'export') this.model.enqueue('export', req, res) - else return res.status(400).json({error: 'Unsupported method'}) - } - - /** - * Deletes a dataset from the cache - */ - this.delete = function (req, res) { - this.model.log.debug(JSON.stringify({route: 'dataset:delete', params: req.params, query: req.query})) - var ids = this.model.decomposeId(req.params.dataset) - this.model.dropResource(ids.item, ids.layer, {}, function (err) { - if (err) return res.status(500).send({error: err.message}) - res.status(200).json({status: 'Deleted'}) - }) - } - - this._getDataset = function (req, res) { - this.findRecord(req.params, function (err, dataset) { - if (err) return res.status(404).json({error: err.message}) - res.status(200).json({dataset: dataset}) - }) - } - - this._getDatasets = function (req, res) { - this.model.findRecords(req.query, function (err, datasets) { - if (err) return res.status(500).json({error: err.message}) - res.status(200).json({datasets: datasets}) - }) - } -} -``` -_[back to top](#contents)_ +See the Koop CLI [documentation](https://github.com/koopjs/koop-cli#add) for additional options and details. diff --git a/_includes/docs_nav.html b/_includes/docs_nav.html index f9b4a4e..5751c1a 100755 --- a/_includes/docs_nav.html +++ b/_includes/docs_nav.html @@ -13,22 +13,23 @@

diff --git a/css/main.scss b/css/main.scss index 69dcb53..ea24762 100755 --- a/css/main.scss +++ b/css/main.scss @@ -149,41 +149,49 @@ h1:hover, .h1:hover, h2:hover, .h2:hover, h3:hover, .h3:hover, h4:hover, .h4:hov border: none; padding-top: 4px; padding-bottom: 4px; + a, a:hover, a:focus { + color: #7d7d7d; + } &.active { font-weight: 500; color: #008cba; - .arrow-wrapper { - .active-icon { - display: inline; - } + a, a:hover, a:focus { + color: #008cba; + } + .sublist-group { + display: block; } } - } - - .sublist-group { - padding-left: 0; - - .list-sub-group-item { - display: block; - list-style-type: none; - padding-left: 10px; - padding-top: 2px; - padding-bottom: 2px; - font-size: 14px; - color: #7d7d7d; + .sublist-group { + padding-left: 12px; + list-style-type: disc; + display: none; &.active { - color: #008cba; - .arrow-wrapper { - .active-icon { - display: inline; - } + display: block; + } + .list-sub-group-item { + display: block; + padding-left: 0px; + padding-top: 2px; + padding-bottom: 2px; + font-size: 13px; + color: #7d7d7d; + font-weight: 300; + + li { + display: block; + } + + &.active { + color: #008cba; + text-decoration: underline; + font-weight: 400; } } } - } - + } } } \ No newline at end of file