How can GitHub Apps use webhooks and APIs together to build reliable real-time integrations? #207726
π·οΈ Discussion TypeQuestion π¬ Feature/Topic AreaAPI BodyIβm learning about GitHub Apps, REST/GraphQL APIs, and webhooks, and Iβm curious about how they should work together in a real-world application. Suppose Iβm building a GitHub App that monitors pull requests, reviews, issues, and repository activity. Webhooks can notify the app when an event occurs, while the API can be used to retrieve additional information. What would be the recommended architecture for handling this reliably? When should I rely on webhook payloads versus making additional API requests? Iβd especially appreciate advice based on real-world GitHub App implementations. |
Replies: 3 comments
|
A good approach is to treat webhooks as the event notification layer and GitHub APIs as the data retrieval layer. When a webhook is received, the application should first verify the webhook signature and event information. The webhook payload should be used directly whenever it contains all the information needed. Additional REST or GraphQL API requests should only be made when more data is required. For reliability, the application should:
REST is usually simpler when the application needs a specific resource or endpoint. GraphQL can be useful when the application needs related data from multiple resources because the client can request the fields it actually needs. A scalable arch |
|
A pattern that has worked well for GitHub Apps is to treat webhooks as the event source and the GitHub API as the source for additional/current data, rather than polling the API continuously. A typical architecture would look something like: GitHub β Webhook endpoint β Validate β Persist/Queue event β Worker β GitHub API β Your DB 1. Use the webhook payload firstI would process as much as possible from the webhook payload itself. Webhook payloads often contain the repository, installation, sender, PR/issue information, action, etc. If the payload already contains everything required for the operation, there is no reason to make another API request. For example, if I receive a I'd make an API request when I need information that isn't included in the payload, need a more complete representation, or need to verify/fetch the latest state. This also helps avoid unnecessary API/rate-limit usage. GitHub itself recommends webhooks over polling when you need to monitor many resources. 2. Make webhook processing idempotentI wouldn't assume webhook delivery is exactly-once or ordered. GitHub documents that webhook deliveries can arrive out of order, and delivery can also be delayed. I'd therefore store the delivery ID ( For example: The worker can then safely retry a job without accidentally performing the same operation twice. 3. Don't do heavy work inside the webhook requestFor a small application, synchronous processing may be fine. As the number of repositories/events grows, I'd put a queue between the webhook endpoint and the actual processing: This prevents a slow GitHub API request or temporary database problem from causing the webhook request itself to fail. GitHub also recommends asynchronous webhook handling with a queue for higher-volume applications. 4. REST vs GraphQLI wouldn't choose GraphQL simply because it's "better." For webhook-driven processing, REST is often the simpler choice because GitHub provides straightforward endpoints for resources such as:
I'd use GraphQL when I need to retrieve several related pieces of data efficiently or want to avoid making many REST requests. So my general rule would be: REST β straightforward resource lookup GraphQL β complex/related data requirements where reducing round trips is valuable The important thing is to measure API usage rather than automatically choosing one. 5. Authentication and rate limitsFor a GitHub App, I'd authenticate API requests using an installation access token appropriate to the installation rather than using a personal access token. Installation tokens expire after one hour, so the application should have a mechanism for obtaining a fresh token when necessary. I'd also:
GitHub App installation rate limits are also tied to the installation and can scale based on repositories/users, so designing around unnecessary API calls becomes increasingly important as the App grows. 6. Webhook securityThe webhook endpoint should not simply trust any POST request. I'd configure a strong webhook secret and verify the The secret should be stored securely and never committed to the repository. GitHub specifically recommends validating webhook signatures before processing deliveries. 7. Handling missed/failed eventsI'd keep an event/delivery record containing things such as: Then failed jobs can be retried independently. I'd also monitor failed webhook deliveries and have a recovery/reconciliation mechanism. GitHub provides APIs to list and redeliver GitHub App webhook deliveries, which can help with operational recovery. For particularly important data, I would also consider periodic reconciliationβfor example, periodically comparing the application's local state with GitHub's current state. That protects you against bugs or events missed while your service was unavailable. 8. The main principleThe design I'd aim for is: Webhook = "Something changed." Webhook payload = "Here's some information about what changed." API = "Give me the additional/current state I need." Queue = "Process this reliably even if my application/API/database is temporarily unavailable." Database = "Remember what I've already processed." This approach keeps the webhook endpoint fast, reduces unnecessary API calls, makes duplicate/out-of-order deliveries manageable, and gives you a much better path to scaling from a few repositories to thousands. For a small GitHub App, you don't necessarily need all of this on day one. I'd start with webhook validation + idempotent processing + installation authentication, then introduce a queue and reconciliation process when the event volume or reliability requirements justify it. |
|
I'd use webhooks as the trigger and the API as the source of additional data. The webhook payload often contains enough information to handle simple events, but I'd fetch the full resource when I need more details or the payload isn't sufficient. For reliability, I'd put incoming events into a queue, acknowledge them quickly, and process them asynchronously. I'd also store the delivery ID to prevent duplicate processing, and use retries with exponential backoff for temporary failures. REST is a good default for straightforward requests, while GraphQL is useful when you need several related resources or want to reduce the number of API calls. For authentication, use GitHub App installation tokens rather than personal access tokens. Verify webhook signatures, respect rate limits, and monitor failed deliveries. As the number of repositories grows, a queue, idempotent handlers, and per-installation rate-limit tracking become increasingly important. |
A good approach is to treat webhooks as the event notification layer and GitHub APIs as the data retrieval layer.
When a webhook is received, the application should first verify the webhook signature and event information. The webhook payload should be used directly whenever it contains all the information needed. Additional REST or GraphQL API requests should only be made when more data is required.
For reliability, the application should: