OCA · RabbitJon-S73 · Aug 4, 2019 · Jul 3, 2021 · Jul 5, 2021 · Jul 5, 2021
diff --git a/bus_alt_connection/README.rst b/bus_alt_connection/README.rst
@@ -0,0 +1,189 @@
+.. image:: https://odoo-community.org/readme-banner-image
+   :target: https://odoo-community.org/get-involved?utm_source=readme
+   :alt: Odoo Community Association
+
+==================
+Bus Alt Connection
+==================
+
+.. 
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! This file is generated by oca-gen-addon-readme !!
+   !! changes will be overwritten.                   !!
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+   !! source digest: sha256:ab7e1b9d5721f8cb27f93c58ed77e5034bc0099105ac6d5097f1bdc74a4e6973
+   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
+    :target: https://odoo-community.org/page/development-status
+    :alt: Beta
+.. |badge2| image:: https://img.shields.io/badge/license-AGPL--3-blue.png
+    :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
+    :alt: License: AGPL-3
+.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fserver--tools-lightgray.png?logo=github
+    :target: https://github.com/OCA/server-tools/tree/19.0/bus_alt_connection
+    :alt: OCA/server-tools
+.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
+    :target: https://translation.odoo-community.org/projects/server-tools-19-0/server-tools-19-0-bus_alt_connection
+    :alt: Translate me on Weblate
+.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
+    :target: https://runboat.odoo-community.org/builds?repo=OCA/server-tools&target_branch=19.0
+    :alt: Try me on Runboat
+
+|badge1| |badge2| |badge3| |badge4| |badge5|
+
+This module makes it possible to use
+`PgBouncer <https://pgbouncer.github.io/>`__ as a connection pooler for
+odoo.
+
+Why isn't odoo's connection pooling good enough?
+------------------------------------------------
+
+Odoo's builtin connection pooling works at process level: each Odoo
+process has its own
+`ConnectionPool <https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L525>`__,
+limited to ``db_maxconn``.
+
+It does the job of re-using open connections available in the pool. But
+it never closes these connections, `unless reaching
+db_maxconn <https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L593>`__.
+
+In practice, we observe that each odoo worker will end up with up to 3
+open connection in its pool. With 10 http workers, that's up to 30
+connection continuously open just for one single instance.
+
+Here comes PgBouncer
+--------------------
+
+PgBouncer will help to limit this number of open connections, by sharing
+a pool of connections at the instance level, between all workers. Odoo
+workers will still have up to 3 open connections, but these will be
+connections to PgBouncer, that on its side will close unnecessary
+connections to pg.
+
+This has proven to help performances on Odoo deployments with multiple
+instances.
+
+It allows you to define how resources should be shared, according to
+your priorities, e.g. :
+
+- key odoo instance on host A can open up to 30 connections
+- while odoo instance on host B, dedicated to reports, can open up to 10
+  connections only
+
+And most importantly, it helps you to ensure that ``max_connections``
+will never be reached on pg server side.
+
+Why is this module needed?
+--------------------------
+
+When configuring PgBouncer, you can choose between 2 transaction pooling
+modes:
+
+- pool_mode = session
+- pool_mode = transaction
+
+If we choose pool_mode = session, then one server connection will be
+tied to a given odoo process until its death, which is exactly what
+we're trying to change. Thus, to release the server connection once the
+transaction is complete, we use pool_mode = transaction.
+
+This works fine, except for Odoo's longpolling features that relies on
+the
+`LISTEN/NOTIFY <https://www.postgresql.org/docs/9.6/static/sql-notify.html>`__
+mechanism from pg, which is `not
+compatible <https://wiki.postgresql.org/wiki/PgBouncer>`__ with that
+mode.
+
+To be more precise, NOTIFY statements are properly transfered by
+PgBouncer in that mode; only the LISTEN statement isn't (because it
+needs to keep the server connection open).
+
+So for the unique "listening" connection per instance that requires this
+statement
+(`here <https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L166>`__),
+we need odoo to connect directly to the pg server, bypassing PgBouncer.
+
+That's what this module implements, by overriding the relevant method of
+the
+`Dispatcher <https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L105>`__.
+
+**Table of contents**
+
+.. contents::
+   :local:
+
+Installation
+============
+
+You don't need to install this module in the database(s) to enable it.
+
+But you need to load it server-wide:
+
+- By starting Odoo with ``--load=web,bus_alt_connection``
+- Or by updating its configuration file:
+
+.. code:: ini
+
+   [options]
+   (...)
+   server_wide_modules = web,bus_alt_connection
+
+Configuration
+=============
+
+You need to define how to connect directly to the database:
+
+- Either by defining environment variables:
+
+     - ``IMDISPATCHER_DB_HOST=db-01``
+     - ``IMDISPATCHER_DB_PORT=5432``
+
+- Or in Odoo's configuration file:
+
+.. code:: ini
+
+   [options]
+   (...)
+   imdispatcher_db_host = db-01
+   imdispatcher_db_port = 5432
+
+Bug Tracker
+===========
+
+Bugs are tracked on `GitHub Issues <https://github.com/OCA/server-tools/issues>`_.
+In case of trouble, please check there if your issue has already been reported.
+If you spotted it first, help us to smash it by providing a detailed and welcomed
+`feedback <https://github.com/OCA/server-tools/issues/new?body=module:%20bus_alt_connection%0Aversion:%2019.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
+
+Do not contact contributors directly about support or help with technical issues.
+
+Credits
+=======
+
+Authors
+-------
+
+* Trobz
+
+Contributors
+------------
+
+- Nils Hamerlinck <[email protected]>
+
+Maintainers
+-----------
+
+This module is maintained by the OCA.
+
+.. image:: https://odoo-community.org/logo.png
+   :alt: Odoo Community Association
+   :target: https://odoo-community.org
+
+OCA, or the Odoo Community Association, is a nonprofit organization whose
+mission is to support the collaborative development of Odoo features and
+promote its widespread use.
+
+This module is part of the `OCA/server-tools <https://github.com/OCA/server-tools/tree/19.0/bus_alt_connection>`_ project on GitHub.
+
+You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.
diff --git a/bus_alt_connection/__init__.py b/bus_alt_connection/__init__.py
@@ -0,0 +1,3 @@
+# Copyright 2019 Trobz <https://trobz.com>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+from . import models
diff --git a/bus_alt_connection/__manifest__.py b/bus_alt_connection/__manifest__.py
@@ -0,0 +1,15 @@
+# Copyright 2019 Trobz <https://trobz.com>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+{
+    "name": "Bus Alt Connection",
+    "summary": "Needed when using PgBouncer as a connection pooler",
+    "version": "19.0.1.0.0",
+    "author": "Trobz,Odoo Community Association (OCA)",
+    "website": "https://github.com/OCA/server-tools",
+    "category": "Extra Tools",
+    "license": "AGPL-3",
+    "depends": ["bus"],
+    "installable": True,
+    "auto_install": False,
+}
diff --git a/bus_alt_connection/i18n/bus_alt_connection.pot b/bus_alt_connection/i18n/bus_alt_connection.pot
@@ -0,0 +1,13 @@
+# Translation of Odoo Server.
+# This file contains the translation of the following modules:
+#
+msgid ""
+msgstr ""
+"Project-Id-Version: Odoo Server 14.0\n"
+"Report-Msgid-Bugs-To: \n"
+"Last-Translator: \n"
+"Language-Team: \n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=UTF-8\n"
+"Content-Transfer-Encoding: \n"
+"Plural-Forms: \n"
diff --git a/bus_alt_connection/models/__init__.py b/bus_alt_connection/models/__init__.py
@@ -0,0 +1,3 @@
+# Copyright 2019 Trobz <https://trobz.com>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+from . import bus
diff --git a/bus_alt_connection/models/bus.py b/bus_alt_connection/models/bus.py
@@ -0,0 +1,67 @@
+# Copyright 2019 Trobz <https://trobz.com>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+import json
+import logging
+import os
+import selectors
+
+import psycopg2
+
+import odoo
+from odoo.tools import config
+
+import odoo.addons.bus.models.bus
+from odoo.addons.bus.models.bus import TIMEOUT, hashable, stop_event
+
+_logger = logging.getLogger(__name__)
+
+
+def _connection_info_for(db_name):
+    db_or_uri, connection_info = odoo.sql_db.connection_info_for(db_name)
+
+    for p in ("host", "port"):
+        cfg = os.environ.get(f"ODOO_IMDISPATCHER_DB_{p.upper()}") or config.get(
+            "imdispatcher_db_" + p
+        )
+        if cfg:
+            connection_info[p] = cfg
+    return connection_info
+
+
+class ImDispatch(odoo.addons.bus.models.bus.ImDispatch):
+    def loop(self):
+        """Dispatch postgres notifications to the relevant
+        polling threads/greenlets"""
+        connection_info = _connection_info_for("postgres")
+        _logger.info(
+            "Bus.loop listen imbus on db postgres (via %(host)s:%(port)s)",
+            connection_info,
+        )
+        conn = psycopg2.connect(**connection_info)
+        with conn.cursor() as cr, selectors.DefaultSelector() as sel:
+            cr.execute("listen imbus")
+            conn.commit()
+            sel.register(conn, selectors.EVENT_READ)
+            while not stop_event.is_set():
+                if sel.select(TIMEOUT):
+                    conn.poll()
+                    channels = []
+                    while conn.notifies:
+                        channels.extend(json.loads(conn.notifies.pop().payload))
+                    # relay notifications to websockets that have
+                    # subscribed to the corresponding channels.
+                    websockets = set()
+                    for channel in channels:
+                        websockets.update(
+                            self._channels_to_ws.get(hashable(channel), [])
+                        )
+                    for websocket in websockets:
+                        websocket.trigger_notification_dispatching()
+
+
+odoo.addons.bus.models.bus.ImDispatch = ImDispatch
+dispatch = ImDispatch()
+odoo.addons.bus.models.bus.dispatch = dispatch
+odoo.addons.bus.models.ir_websocket.dispatch = dispatch
+odoo.addons.bus.websocket.dispatch = dispatch
diff --git a/bus_alt_connection/pyproject.toml b/bus_alt_connection/pyproject.toml
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["whool"]
+build-backend = "whool.buildapi"
diff --git a/bus_alt_connection/readme/CONFIGURE.md b/bus_alt_connection/readme/CONFIGURE.md
@@ -0,0 +1,15 @@
+You need to define how to connect directly to the database:
+
+- Either by defining environment variables:
+
+  > - `IMDISPATCHER_DB_HOST=db-01`
+  > - `IMDISPATCHER_DB_PORT=5432`
+
+- Or in Odoo's configuration file:
+
+``` ini
+[options]
+(...)
+imdispatcher_db_host = db-01
+imdispatcher_db_port = 5432
+```
diff --git a/bus_alt_connection/readme/CONTRIBUTORS.md b/bus_alt_connection/readme/CONTRIBUTORS.md
@@ -0,0 +1 @@
+- Nils Hamerlinck \<<[email protected]>\>
diff --git a/bus_alt_connection/readme/DESCRIPTION.md b/bus_alt_connection/readme/DESCRIPTION.md
@@ -0,0 +1,71 @@
+This module makes it possible to use
+[PgBouncer](https://pgbouncer.github.io/) as a connection pooler for
+odoo.
+
+## Why isn't odoo's connection pooling good enough?
+
+Odoo's builtin connection pooling works at process level: each Odoo
+process has its own
+[ConnectionPool](https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L525),
+limited to `db_maxconn`.
+
+It does the job of re-using open connections available in the pool. But
+it never closes these connections, [unless reaching
+db_maxconn](https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L593).
+
+In practice, we observe that each odoo worker will end up with up to 3
+open connection in its pool. With 10 http workers, that's up to 30
+connection continuously open just for one single instance.
+
+## Here comes PgBouncer
+
+PgBouncer will help to limit this number of open connections, by sharing
+a pool of connections at the instance level, between all workers. Odoo
+workers will still have up to 3 open connections, but these will be
+connections to PgBouncer, that on its side will close unnecessary
+connections to pg.
+
+This has proven to help performances on Odoo deployments with multiple
+instances.
+
+It allows you to define how resources should be shared, according to
+your priorities, e.g. :
+
+- key odoo instance on host A can open up to 30 connections
+- while odoo instance on host B, dedicated to reports, can open up to 10
+  connections only
+
+And most importantly, it helps you to ensure that `max_connections` will
+never be reached on pg server side.
+
+## Why is this module needed?
+
+When configuring PgBouncer, you can choose between 2 transaction pooling
+modes:
+
+- pool_mode = session
+- pool_mode = transaction
+
+If we choose pool_mode = session, then one server connection will be
+tied to a given odoo process until its death, which is exactly what
+we're trying to change. Thus, to release the server connection once the
+transaction is complete, we use pool_mode = transaction.
+
+This works fine, except for Odoo's longpolling features that relies on
+the
+[LISTEN/NOTIFY](https://www.postgresql.org/docs/9.6/static/sql-notify.html)
+mechanism from pg, which is [not
+compatible](https://wiki.postgresql.org/wiki/PgBouncer) with that mode.
+
+To be more precise, NOTIFY statements are properly transfered by
+PgBouncer in that mode; only the LISTEN statement isn't (because it
+needs to keep the server connection open).
+
+So for the unique "listening" connection per instance that requires this
+statement
+([here](https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L166)),
+we need odoo to connect directly to the pg server, bypassing PgBouncer.
+
+That's what this module implements, by overriding the relevant method of
+the
+[Dispatcher](https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L105).