Integration Patterns: Connecting Data Sources in Enterprise Architecture

Data integration is the process of combining data from disparate sources — operational databases, SaaS platforms, file exports, IoT sensors, partner systems — into a unified view that supports analytical and operational workloads. The pattern chosen for each integration relationship determines the latency, reliability, scalability, and coupling between systems.

Enterprise data architectures in large Canadian organizations typically involve dozens or hundreds of integration points spanning internal systems, cloud platforms, and external partners. Managing this complexity requires a systematic understanding of integration patterns and the trade-offs each represents, rather than ad hoc point-to-point connections that create fragile, unmanageable dependency webs.

Event-Driven Integration

Event-driven integration treats state changes in a source system as discrete events that are published to a shared event stream. Downstream systems subscribe to relevant events and process them independently, without direct coupling to the source system. This approach enables loose coupling between systems — the source does not know or care which consumers are processing its events — and supports high scalability through the parallelism of independent consumers.

Event Streaming Platforms

Apache Kafka is the dominant event streaming platform in enterprise deployments. Kafka organizes events into named topics, with each topic partitioned for parallelism and replicated for durability. Producers publish events to topics; consumers maintain their own read offset within each partition, enabling independent, resumable consumption. Confluent Cloud and Amazon MSK (Managed Streaming for Apache Kafka) are commonly used managed Kafka services for organizations that prefer to avoid self-managing Kafka infrastructure.

Azure Event Hubs and Amazon Kinesis are alternative managed event streaming services that offer a subset of Kafka's capabilities with tighter integration into their respective cloud ecosystems. Organizations choosing between Kafka-based and native cloud streaming services should evaluate protocol compatibility with existing tooling, cross-cloud portability requirements, and the operational maturity of the team.

Event Schema Management

Event schema compatibility is a critical operational concern in event-driven architectures. As producer systems evolve, events may change in ways that break downstream consumers. A schema registry — such as Confluent Schema Registry, AWS Glue Schema Registry, or Apicurio — enforces schema compatibility rules (backward compatible, forward compatible, or full compatible) at publish time, preventing incompatible schema changes from reaching consumers. Apache Avro is the most common serialization format used with schema registries due to its compact binary encoding and embedded schema evolution semantics.

API-Based Integration

API-based integration connects systems through request-response interfaces, most commonly REST APIs or GraphQL endpoints. This pattern is appropriate when data must be retrieved on demand, when access requires authentication and authorization enforced by the source system, or when the volume of data per request is small relative to the overhead of establishing the connection.

REST APIs are the dominant integration interface for SaaS platforms, partner data exchanges, and public data sources. Most analytics platforms and ETL tools (Fivetran, Airbyte, Stitch) provide pre-built connectors for common SaaS APIs, abstracting the authentication, pagination, and rate-limiting logic that would otherwise need to be implemented per-integration.

API-based integration introduces dependencies on source system availability and rate limits. Caching strategies (storing API responses for a defined period before re-fetching) and circuit breaker patterns (pausing requests to a failing API to prevent cascade failures) are important reliability mechanisms for API-dependent pipelines.

File-Based Integration

File-based integration exchanges data through files deposited in shared storage locations — SFTP servers, S3 buckets, Azure Blob Storage containers, or network file shares. Despite its simplicity, file-based integration remains widely used in enterprise environments, particularly for batch integrations with legacy systems, partner organizations, and government entities that may not provide API access.

Common file formats for enterprise integration include CSV, fixed-width flat files, JSON, XML, and Parquet. The choice of format is typically dictated by the producing system's export capabilities and the consuming system's ingestion requirements. Schema enforcement is the primary challenge with file-based integration: files may arrive with inconsistent column orders, missing headers, encoding variations, or undocumented format changes.

Effective file-based integration pipelines implement validation gates that inspect incoming files before processing — checking for expected column presence, data type conformance, row count plausibility, and file completeness indicators. Files that fail validation are quarantined and trigger alerts rather than being silently ingested with quality issues.

Synchronous vs. Asynchronous Integration

Synchronous integration requires the requesting system to wait for a response before proceeding. REST API calls are synchronous by default: the caller sends a request and blocks until the response arrives. Synchronous integration is simple to reason about but creates temporal coupling — the caller is unavailable while waiting — and is sensitive to source system latency and availability.

Asynchronous integration decouples the sender from the receiver through an intermediary (a message queue, event stream, or file store). The sender deposits a message or file and continues without waiting for the receiver to process it. Asynchronous patterns improve resilience — the receiver can be temporarily unavailable without blocking the sender — but introduce eventual consistency, where the receiver's state may lag the sender's state by the time required to process the backlog.

The choice between synchronous and asynchronous integration is driven by the latency requirement of the consuming use case and the availability and scalability constraints of the systems involved. Operational workflows that require immediate confirmation (inventory reservation, payment processing) necessitate synchronous integration. Analytics pipelines, which tolerate minutes to hours of latency, benefit from the resilience of asynchronous approaches.

Change Data Capture

Change Data Capture (CDC) captures row-level changes (inserts, updates, deletes) from a source database's transaction log and publishes them as a stream of change events. CDC enables low-latency replication of database changes without polling the source table, which avoids the performance impact and detection gaps of query-based incremental extraction.

Debezium is the most widely deployed open-source CDC tool, supporting source connectors for PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB. CDC change events are typically published to Kafka topics and consumed by downstream pipelines that apply the changes to a destination database or data lake. Managing deletes correctly — applying delete events from the source to the destination rather than ignoring them — is a common point of implementation error in CDC pipelines.