Backend Synergies: Seamless 3rd-Party Integrations

The term “integration” can mean a lot of different things. In fact, integrating a CRM with other services is possible on many different layers and with many different techniques. Let’s explore some common use cases:

• Form integration
  A common use case is the integration of forms. The CMS displays a predefined form to a user and sends the filled-out form to an external system for further processing. In an advanced variant, the external system dynamically generates the form, sends it to the CMS, and gets the filled-out form back.
  A typical flow for form integration is used for signing up for newsletters. Typically, the user clicks a button or checks a checkbox to confirm their interest in subscribing, and the CMS then displays an embedded subscription form provided by an external mailing service like Mailchimp.
• Imports
  Many scenarios require a simple one-way transfer of semi-static data, like product catalogs or event listings, from an external system to the CMS. These imports can occur at regular intervals or on-demand, depending on the data update frequency.
• Live synchronization
  The supreme discipline of CMS integration is live synchronization between the CMS and an external service. A perfect example of this is the integration of Sylius, a headless e-commerce framework, with Sulu. Integrating a headless system with a CMS frontend requires data to travel between the systems in real time, requiring a fast and reliable connection.
• Single Sign-On
  If your daily work includes using several independent applications, a Single Sign-On (SSO) provider can reduce the number of required logins to one. All applications integrated with a common SSO service share a user’s login status.

Every use case has particular requirements. A good CMS system, therefore, supports a number of different integration techniques that we’ll explore next.

Sulu supports a range of integration approaches, depending on the kind of data, how the data is processed, and how frequently it is being exchanged between the systems.

APIs and Webhooks

An Application Programming Interface (API) is a rather straightforward way of interacting with another application. Exposing APIs allows external clients to interact with your CMS directly, enabling data retrieval, updates, and triggering computations. Conversely, your CMS can consume third-party APIs for services like image resizing or data enrichment. Webhooks provide a simplified variant for receiving events from remote systems.

In Sulu, for example, you can use Symfony’s HTTP client to call external REST APIs.

The reverse direction is also possible. In fact, exposing a REST API takes minimal effort. A REST API request is technically an HTTP request, and handling HTTP requests is a CMS’s bread and butter. So to expose an API endpoint with Sulu, you would only need to implement a custom controller to process the request.

Event dispatchers and listeners

Symfony’s event system is an excellent way of passing data or invoking tasks asynchronously. While Symfony events are not designed to be sent to remote systems, they can still serve to integrate applications that run on the same Symfony stack.

Sulu is a Symfony application and, therefore, can be integrated with other applications via Symfony bundles. Symfony applications could expose PHP APIs to call each other directly, but this can lead to a very tight coupling between those applications. Tight couplings cause downstream issues like high maintenance costs. Symfony’s event system allows for a loose coupling between applications: The sender simply dispatches events to a queue, and the receiver listens on the queue for these events. Neither the sender nor the receiver needs hard-coded knowledge about each other’s internals.

Sulu uses the event system to send events like PageCreated, PageModified, PagePublished and so on. Custom EventListener or EventSubscriber services can listen to and act on these events.

Message queues

Symfony events are great for passing messages inside a Symfony stack. Message queues expand this model to distributed systems. A message queue is a medium for transporting commands or data between remote applications in a fail-safe way. Producers and consumers connect to the queue to publish or receive messages.

Examples of open-source message queue systems include Redis (with Redis Streams) RabbitMQ, NATS, and Apache Kafka. The Symfony Messenger enables Symfony applications to connect to remote systems via one of the various asynchronous message queues Messenger can connect to (like AMQP, SQS, Redis).

Sulu is already highly scalable. Message queues extend this scalability to the applications and infrastructure that the CMS interacts with.

Scheduled imports

When a CMS exposes external data that changes at a slow rate, a scheduled import is a simple and convenient way to incorporate changes into the CMS. Consider the example of event listings—cinemas, exhibitions, concert halls, and festivals announce their programs way ahead of time, so a CMS that lists those events might only need to query and import new programs and announcements once per day.

Such data is imported to either the CMS’s database, or into the file system, or a combination of both. Hence, in a typical scenario for a scheduled import, the Symfony Scheduler could invoke a Symfony service in regular intervals that gathers data, formats it, and adds it to a dedicated database table where the CMS can retrieve it.

Real-time Synchronization

A two-way live synchronization of data between Sulu and external systems can be managed at various levels.

• At the database and file system level. If external data is updated independently of user requests, it is reasonable to synchronize data directly to and from the database or the file system. The database is especially suitable for this, thanks to the transaction mechanism that preserves data consistency in the presence of concurrent updates.
• At the business logic level. Data synchronizations that are triggered from user requests are best handled by custom controllers and services.

Single Sign-On with OpenID

As discussed above, Single Sign-On (SSO) enables a user to log in to several applications with a single login step. From version 2.6 onwards, Sulu provides SSO over OpenID Connect (OIDC), an authentication protocol based on the authorization framework OAuth 2.0. Companies or agencies who have accounts at Google Business, Microsoft, or AWS can start using OIDC right away. Moreover, it is possible to connect a self-hosted Keycloak server to Sulu Admin.

Besides picking the integration method that is most suitable for your particular use case, there are a few other aspects to consider. The following list is not exhaustive and shall serve as a starting point. Contact us and we can help you uncover all the aspects you need to consider for your specific project.

Handling live data

Live data is data that changes frequently, whether the change happens slowly (like, for example, weather data) or in real-time (such as a video stream from a webcam).

For such cases of routing live data to the web, you will almost always want to write a custom controller that handles the constant stream of data as it occurs. Other methods, like saving a data stream to the database or a caching service and serving it from there, will add unnecessary latency to the flow.

Use message queues for robust communication

Communication between distributed systems has many possible sources of failure, from network issues to crashing services or shut-down nodes. Message queues like RabbitMQ, NATS, or Kafka are designed to deal with such unreliabilities. They implement automatic retries, persistence of messages, deduplication, and more. Message queue systems either provide a specific delivery promise or even let you choose among different delivery promises, which include:

• At-least-once delivery: Messages are delivered once or multiple times, but they never get lost.
• At-most-once delivery: The queue may lose messages, but it will never deliver duplicates of the same message.
• Exactly-once delivery: The queue never loses a message and never delivers a message twice.

Picking a delivery promise can help you find the balance between speed and reliability that suits your needs.

Caching of external information

Retrieving information from external sources can be time-consuming. If you know that the retrieved data will be reused across requests, store it in a local cache inside the CMS. This cache can be the database, a centralized, in-memory cache like Redis, or an HTTP cache.

How to handle errors

Communication between systems can fail for various reasons, and errors are best handled at the place where they occur first.

Possible integration errors can occur at various points:

• When calling an external API.
  An external API (or a corresponding SDK, if available) flags possible issues by returning errors, status codes, or other indicators.
• On web page level.
  Frontend code may call into third-party APIs and hence must be equipped to deal with errors gracefully. This is particularly important for scenarios like partial page updates that fetch data from external systems.
• In a Symfony Messenger transport.
  Symfony Messenger allows the use of asynchronous transports, where a sender can file a message to send and continue with other things without having to wait for the result. Handling errors from sending asynchronous messages is, therefore, a bit more involved than just checking an exception or an error returned by a function call. Fortunately, Messenger provides a means for implementing retry strategies and failure transports to store error messages for later processing.

TIP: To manage errors that aren’t caught by the application, review the var/log directory. Integrating error tracking platforms like Sentry in your PHP code or with Symphony even allows you to get notified about runtime errors as they occur.

When all our applications are powered by AI, we’ll be able to just tell them to talk to each other to figure out how to process that user form or get that dynamic content out to the website. But we’re not there yet.

Let Sulu take you into the future, with plenty of tested and proven (hallucination-free!) ways to bring your digital systems together. We can help you select the most suitable integration technique for your particular integration requirements. Get in touch!