Description

This PR introduces a new communication architecture for our microservices-based system, using Redis Pub/Sub. The system is designed to be extensible with a plugin system, where each plugin can act as a publisher, subscriber, or both, and interact with other services via Pub/Sub. At this stage, only the subscriber plugin has been implemented.

Related Issue: N/A (if applicable, add the issue number here)

Motivation: The main motivation for these changes is to give users the opportunity to customize the workflow and create a plugin ecosystem. The new architecture provides greater flexibility for users to extend the system with their own functionality, and the use of Redis Pub/Sub allows for more efficient and scalable communication between plugins.

System Operation

Here is a simplified explanation of how these microservices run:

1. Initialization: Each microservice, when started, initializes a connection with a central Redis server. The services could be either publishers, subscribers, or both, depending on their roles. If they are publishers, they are assigned a priority that dictates the order in which they can publish messages.

2. Publishing: If a service is a publisher, it can create messages and publish them to a specific topic. The message is sent to the Redis server, which handles distributing the message to all subscribers of that topic.

3. Subscribing: If a service is a subscriber, it listens to one or more topics on the Redis server. When a message is published to a topic it is subscribed to, the Redis server sends the message to the service.

4. Processing and Republishing: Once a service receives a message, it processes the message based on its internal logic. If the service is both a subscriber and a publisher, it may modify the message and republish it to the same topic. The modified message is then passed down to the next lower priority service in the chain for further processing. This functionality is not yet implemented.

5. Scaling: This architecture supports scaling by adding more instances of services. Each instance would have its own connection to the Redis server and can independently publish and/or subscribe to messages.

6. Plugin System: The architecture is designed to be extensible with a plugin system. Each plugin is essentially a microservice that can publish, subscribe, or both. The system allows for easy addition or removal of plugins, providing a lot of flexibility in customizing the workflow.

All services and plugins are Docker-based containers, which allows them to be written in any language that has a Redis SDK, providing even more flexibility.

Pending Features and Requests for Feedback

• Implementation of publisher plugins and the "pass to the lower system" functionality is pending.
• Test coverage has not yet been implemented, as we are seeking feedback on the current feature set first.
• Suggestions on how to better structure the system would be greatly appreciated.

Checklist before requesting a review

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged

Screenshots (if appropriate):

N/A

Please review these changes and provide feedback.