Understanding API Integrations Between Business Systems

Follow one piece of data and you will find the whole problem

Take an order that arrives in a business that grew organically. The salesperson typed it into a spreadsheet and emailed it. Someone in operations re-keys it into an ERP system. Finance exports the invoice register to a file every morning and reconciles it against the bank feed by hand. Support looks up the customer in a CRM that has a different phone number for the same person, because nobody has agreed which system owns contact details.

Every one of those steps is a small decision somebody made once. None of them is unreasonable in isolation. Together they add up to a business that cannot answer simple questions about its own customers, cannot reliably tell whether an order has been paid for, and cannot onboard anybody in less than an afternoon of waiting on other teams.

This is what an integration project is actually for. It is not about connecting systems; it is about making one system the owner of each piece of information, and removing the copies. That distinction is the whole argument of this article, because most integration failures are data ownership failures wearing a technical disguise.

Decide who owns each piece of data before writing any code

Before an integration gets specified, somebody has to answer: for this field, which system is the source of truth, and what happens when they disagree? Not what should happen ideally. What actually happens, including on the days when somebody updated the wrong record in the wrong system.

The answers look obvious in isolation, which is exactly why they need writing down. The customer’s postal address is usually the authority. Whether the sales pipeline is the authority for the account owner is a question with no default answer, and it changes what the CRM is allowed to overwrite when a sync runs. Where two systems both maintain a value, the integration has to pick, and picking silently produces the situation where the same customer appears with two different addresses depending on which screen you opened.

Our API integration work starts with a table: field, owning system, direction of flow, and what happens on conflict. That table is usually more contentious than the technical design, and it is where the project is won or lost. Once it is agreed, a surprising amount of the implementation becomes mechanical. SmartEdge IT Solutions keeps the table in the same repository as the integration code, because the version in a project document is stale within a quarter and the one in the repository gets updated by whoever changes the code.

Two rules make the difference afterwards.

• Never overwrite a master record from a system that does not own it. A read-only integration is a valid design choice, and choosing one removes a class of bug entirely.
• Record where every value came from. Provenance on a field is what makes a discrepancy diagnosable in five minutes instead of an afternoon. When the support team says the phone number is wrong, the first question is which system last wrote it.

Pick a pattern per connection, not per project

Each integration between two systems has one of a few shapes, and choosing the wrong one is the most common design mistake.

Synchronous request and response

The calling system needs an answer immediately. Validating a customer postcode while somebody is still filling in a form, checking stock before a sale is confirmed. The cost is that availability of one system becomes availability of both, and a timeout leaves the user staring at a spinner. Timeouts, retries with backoff and a defined behaviour when the other side is down all have to be designed, not invented during the first incident.

Scheduled batch

Data moves on a schedule in both directions. Nightly invoice reconciliation, a daily stock position, a feed of new accounts into an accounting system. This is the pattern that fits most reporting and finance work, and it is far easier to reason about than anything real-time. Its weakness is latency: between the schedule, the two systems disagree, and somebody will eventually ask why the number on screen does not match the number in the report.

Within a batch, two sub-decisions matter more than people expect. Incremental updates, so a nightly job transfers only what changed since last night rather than everything, because full transfers stop working quietly once the data is large enough. And a reconciliation step, so a batch can report what it transferred, what it could not transfer, and what it deliberately skipped.

Events and messaging

When one system needs to tell several others that something happened. Order placed, invoice paid, customer record updated. Producers publish; consumers subscribe and do their own work at their own pace. This decouples the systems properly and it is the right answer for fan-out, but it introduces a message broker, delivery semantics to think about, and the need to handle messages arriving more than once or not at all.

Choosing between these is mostly a question of whether the receiving side needs an answer before the user does something. If yes, synchronous. If no, batch or events. Most connections are simpler than teams first assume, and a project that mixes all three patterns in the first fortnight has usually over-designed something.

What goes wrong in practice

The failures repeat. It is worth naming them because each has a specific, unglamorous mitigation.

Rate limits and pagination. The remote system caps how fast you may call it. Ignore that and your integration starts producing incomplete data — a customer list missing entries, an import that appears to succeed. Every integration should respect documented limits, page through results rather than assuming a default page size is the whole set, and record how many records it expected against how many it got.

Authentication expiry. Tokens have a lifetime, and a job that runs at four in the morning finds out when the token expired four hours ago. Handle expiry deliberately: refresh proactively where the remote system supports it, and re-authenticate once on an authorisation failure rather than giving up. Long-running jobs need their token kept alive for the duration rather than fetched once at the start.

Partial failure. The fifth of four hundred records fails validation and the whole job is retried, failing at the same place forever. Per-record error handling, with the failures written somewhere a human will look, is what distinguishes an integration you can operate from one you cannot.

Silent data loss. A field the remote system stops returning is treated by the code as null, and the local record is overwritten with an empty value. Defensive mapping — only writing fields the response actually contained — prevents a class of data loss that can be discovered months later.

Nobody watching it. The most common cause of a long outage is not the failure but the silence. An integration that runs at night and reports nothing has a failure rate that would be alarming if anybody were counting. Log each run with counts and timings, alert on repeated failure, and put a health indicator somewhere a person looks.

Credentials spread around. Shared logins, API keys pasted into scripts, a personal access token belonging to somebody who left. Every remote system should have its own credentials for a named integration, stored in one place, with a named owner and an expiry you can check.

Where the systems come from matters more than the technique

Integrating two systems you control, both with documented APIs and modern authentication, is a week of careful work. Integrating with an on-premise system that predates APIs is a different project entirely, and the difference should be established before anyone quotes.

The realistic options for an old system are these, and they are less interchangeable than they look.

• A database-level integration. Works when you can query its tables. It depends on internal structure, carries none of the old system’s business validation, and breaks when it is upgraded.
• A thin wrapper around its screens. Unpleasant to build and worse to maintain, but occasionally the only route to a capability that is not exposed any other way.
• Modernising it. Fixes the underlying problem properly, costs the most, and takes longest — and produces an asset rather than a workaround.
• Rebuilding the function somewhere else. Add the capability to a system you control, then withdraw the old path as usage moves across.

Each of those is legitimate, and choosing the database-level option when the old system will be replaced within two years is often the correct financial decision even though it is the least satisfying one. What should not happen is choosing it silently and describing it later as an integration. Where the surrounding system is an accounting or operational core, our ERP development work usually begins by establishing what the existing instance can actually expose, because that answer determines the entire approach.

The same question applies in the other direction. A new application that needs to read from the CRM should use the CRM’s supported interface, not a nightly database dump. Our CRM development work treats those integration points as part of the design rather than as something to bolt on once the main features are done, which is cheaper and produces fewer surprises.

Testing an integration is a different discipline

An integration cannot be tested by calling a live third-party system on every run. That makes the suite slow, flaky and occasionally expensive, and it fails in ways that have nothing to do with your code.

The workable approach is layered. Contract tests verify that your request and your handling of the response still match what the remote system documents; these run on every commit and catch the case where somebody assumed a field was called something else. Recorded responses, taken from a real interaction and captured with sensitive data removed, let you test the transformation logic without a network call. A small number of real checks against a sandbox or a low-traffic account confirm that the assumption about the remote system is still true.

Idempotency deserves specific attention. A job that runs twice must not create two orders. This means every write carries a key that the remote system recognises, or every run records what it has already processed. Without it, retrying after a partial failure — which you must be able to do — becomes a source of duplicated records.

Where the automation genuinely helps, and where it does not

There is a temptation to describe integration as a step towards full automation, and it is worth resisting in places. Integrating two systems does not remove the manual step between them unless somebody explicitly designs for it. The invoice reconciliation above still needs a person to look at the exceptions; wiring the systems together makes their job better or worse, and it does not decide which.

What integration does buy is reliability of the data going in and out. Where the wider goal is reducing the manual work around the systems rather than connecting them, that is a different conversation, and it usually has more to do with the process than with the technology. Business process automation work is a reasonable place to start if the manual step, rather than the data transfer, is the thing that hurts. Where the underlying systems have no usable interface at all, adding an integration layer on top of them is a way of making the constraint more visible rather than removing it.

A workable order of work

If a business is where the scenario at the top of this article describes, the sequence that tends to work is unglamorous. Agree the ownership table for the two or three fields that cause the most arguments, because resolving a dispute is often worth more than a new connection. Fix the credentials and the monitoring before adding anything new, since without those, every subsequent integration is invisible. Then pick one connection that removes a measurable amount of manual work and do it properly, including its failure handling.

The temptation is to do all of them at once. It produces a project that is entirely dependent on five vendors behaving predictably and has no intermediate value to show for it, and if one of them is difficult you will have spent the whole budget without finishing. One connection done properly, with a documented field map and a named owner, gives the team the template for the rest and gives the business a reason to fund the rest — and at SmartEdge IT Solutions that has been the pattern that survives contact with a fixed budget more often than the comprehensive one.