A salesperson is on a call, promising a delivery date. The number on their screen comes from Salesforce. If that number was last refreshed by a batch job that ran at 2am, the promise is built on data that's already up to twelve or twenty-four hours stale. Nobody notices until the customer calls back asking why the order that was "confirmed" yesterday doesn't show up anywhere. That's the failure mode that pushes teams from batch to events — not a slide about digital transformation, a specific broken promise made in front of a customer.
Why events, not batch
Batch integration is fine for a long time. Pull a delta every night, upsert it, move on. It's simple to build, simple to reason about, and for most back-office data nobody cares whether it's six hours old. The problem starts the moment the business begins making real-time commitments against that data — a delivery promise, an available-to-sell quantity, a support agent quoting an account balance. At that point staleness isn't a rounding error, it's a lie the system is telling on the business's behalf.
The event-driven shift is a change in who initiates the conversation. In a batch world, Salesforce (or the integration layer) asks the source system: "what changed since last time?" In an event-driven world, the source system doesn't wait to be asked — the moment a fact becomes true, it publishes it. The integration's job changes from polling to reacting.
Two properties of that reaction matter more than anything else in the design that follows. First, ordering: if two events for the same entity arrive close together, they have to be processed in the order they happened, or the last-write-wins logic downstream will happily apply them backwards. A FIFO guarantee — same-entity events processed in publish order — removes an entire category of "why did the record revert" bugs. Second, delivery semantics: most brokers guarantee at-least-once delivery, not exactly-once. That's a deliberate trade-off on the broker's part, and it means the same event can, and eventually will, be delivered twice. If the consumer isn't built to handle that gracefully, you don't get an integration — you get a random number generator. More on how to survive that later.
Three layers, one job each
API-led connectivity gives you a vocabulary for splitting the work — Experience, Process, System — but the names alone don't tell you much until you've seen what each layer actually refuses to do. In an event-driven flow, the three layers map cleanly onto the path an event takes from the broker to Salesforce.
Experience sits closest to the broker. Its only job is to take the raw payload — whatever envelope format, headers and encoding the broker happens to use — and turn it into the shape the rest of the system agreed on internally. It knows the broker's quirks so nothing else has to.
Process is the only layer that knows what an event means for the business. It decides whether this event matters right now, what needs to be enriched before it's usable, and in what order the downstream calls need to happen. It orchestrates; it doesn't talk to Salesforce or to any external API directly.
System is where the actual protocol conversations happen. It speaks the Salesforce connector's language and the enrichment API's language — field names, object types, authentication, pagination. It applies zero business logic; it just executes what Process asked for, faithfully and idempotently.
Simplicity is refusing work
Writing down what each layer does is the easy half of the exercise. The half that actually matters — the one that determines whether this architecture is still coherent in two years — is writing down what each layer is not allowed to do.
- Experience refuses to know business rules. It cannot decide whether an event is worth acting on. If a filtering decision creeps into this layer, you've just made a business rule invisible to everyone who isn't reading broker-adapter code.
- Process refuses to know Salesforce field names. It works entirely in the domain model — customer, order, entitlement — never in API names. The moment a field name from a Salesforce object shows up in an orchestration script, Process has quietly become coupled to System.
- System refuses to transform. If a payload needs reshaping before it's written, that reshaping happens upstream, in Process. System's DataWeave should read almost like a direct field mapping — because if it doesn't, it's carrying logic that belongs somewhere else.
This discipline pays for itself in exactly the moments that used to be expensive. When a Salesforce admin renames a field, one layer changes. When the broker's team changes the envelope format — adds a header, renames a metadata key — one layer changes. Nothing else in the flow needs to know either of those things happened.
A layer that does two things will eventually be changed for two reasons. That is the whole argument.
Five patterns that survive change
Beyond the layering itself, a handful of specific patterns are what separate an integration that ages well from one that gets quietly rewritten eighteen months in.
Contracts published before code
Write the OpenAPI or RAML spec for each layer's interface and publish it — to Anypoint Exchange or wherever your team keeps them — before the implementation is finished. Consumers of that interface build against the contract while the implementation underneath is still moving. The spec becomes the negotiation artifact between teams instead of a Slack thread nobody can find six months later.
Environment-scoped properties with encrypted secrets
Secure properties, encrypted at rest, with the decryption key injected per environment at runtime. No credential — API key, connected app secret, broker password — ever lives in the repository in readable form. Promoting a flow from one environment to the next changes a property file, not a line of application code, which is exactly the boundary you want between "configuration" and "logic."
Idempotent upsert by external ID
This is the single most important pattern once you've accepted at-least-once delivery as a fact of life rather than an edge case. Upserting a Salesforce record keyed on a stable external ID means that replaying the same event twice produces the same row, not two rows and not a duplicate-detection headache. It's what turns "the broker redelivered this message" from an incident into a non-event. Every System-layer write that touches Salesforce should be built this way by default, not as an afterthought bolted on after the first duplicate-record ticket.
Per-record fault tolerance
In any batch — including the reconciliation batch discussed below — one malformed record must never be allowed to roll back the other nine thousand good ones. The naive implementation processes a batch as one transaction and dies on the first bad row. The pattern that survives production isolates each record's failure, logs it with enough context (the source ID, the payload, the error) to replay it later, and keeps the batch moving. A failed record becomes a queue entry, not an outage.
Categorised logging
Every log line carries a category and a correlation ID that follows a single business event across all three layers. "What happened to this specific order" should be answerable with a log query filtered on that correlation ID, not an afternoon spent grepping through three separate runtime logs trying to line up timestamps. Observability that's designed into the flow from day one costs almost nothing; observability bolted on after the first production incident costs a rewrite.
The hard calls
Everything above is close to a solved problem — well-trodden, low-controversy. The decisions below aren't. Each one has a real cost on both sides, and the "right" answer depends on specifics no generic playbook can give you.
Connector vs hand-written DataWeave
The Salesforce connector handles authentication, retry behaviour, batching and API version drift for you, for free, as long as you stay inside what it was designed to do. The default should always be: use the connector. You reach for hand-written DataWeave against the REST or Bulk API only when the mapping is genuinely non-trivial, or when the connector's built-in behaviour hides something you specifically need to control — a retry policy that's wrong for your volume, a batching strategy that doesn't fit your shape of data. The failure mode shows up in both directions: fighting the connector for hours to make it do something it was never built for, or hand-rolling three hundred lines of DataWeave to solve a problem the connector already solved correctly.
Where the encryption boundary sits
Encrypting at the property level keeps the secret out of the repository entirely — but the runtime process still holds the decryption key in memory, and that's a boundary you have to trust. Encrypting at the payload level — individual fields inside the message — closes that gap, but you pay the cost of that encryption at every single hop the payload passes through, and debugging gets meaningfully harder when you can't read a log line without a decryption step. The rule of thumb that holds up: encrypt credentials at rest as secure properties, always. Encrypt payload fields only when a specific regulation actually requires it for that data — not as a default posture "just in case."
Tolerating a 404 from the enrichment API
Here's the scenario: Salesforce knows about an entity — a customer, an account — that the external enrichment API has never heard of. That's a completely normal state, not a data-quality failure. Treat it as an error and you stop the flow for something that happens routinely. Treat it as silence — swallow the 404 and move on — and you've lost the signal that this record needs attention. The answer that actually works is a third option: proceed with the record un-enriched, mark it as such, and let it surface again on the next reconciliation pass. That's precisely why, in the reference architecture above, the enrichment call is drawn as best-effort rather than as a hard dependency in the critical path.
Why you still need the batch
Real-time events are the fast path, not the whole path. A nightly reconciliation job stays necessary for reasons that have nothing to do with how well the event pipeline is built:
- Events get lost. Broker retention windows expire, a consumer is down during a deploy, a message ends up parked in a dead-letter queue and nobody's watching it that day.
- Best-effort enrichment sometimes fails and stays failed. The record from the 404 scenario above needs a second chance to pick up the enrichment it missed the first time.
- The source system gets corrected out-of-band. Someone fixes a record directly in a source database, or through a process that doesn't emit an event at all. The event stream never sees that change; the batch is the only thing that will.
Frame it this way: the event stream gives you speed, the batch gives you correctness. Anyone who tells you real-time events remove the need for reconciliation hasn't run one of these systems in production for a full year — the batch is what quietly catches everything the fast path was never going to see.
When this is overkill
None of this is free. This pattern costs three separate deployables to build, test and operate, a broker to run and monitor, a contract-publishing discipline the team has to actually keep up, and an observability setup that someone has to maintain. Don't pay that bill if:
- Volume is low and hourly — or even daily — staleness is genuinely fine for the business. A single scheduled job is a much smaller thing to own.
- There's exactly one source and one destination with a stable, simple mapping. The layering exists to absorb change from multiple directions at once; with one source and one destination, it's not buying you anything yet.
- The team has no real appetite for operating a broker. A pattern nobody wants to run at 3am is a pattern that will be abandoned at 3am.
It earns its cost back when there are multiple sources feeding a single destination, when the business is making live commitments on top of the data, or when the roadmap already has more systems queued up to join the flow. In those cases the layering isn't ceremony — it's the only thing standing between you and a rewrite eighteen months from now.
If you're weighing whether your own integration needs this shape or something simpler, happy to book a conversation and think it through together.