Building accounting integrations: A guide based on Codat’s experience
A couple months ago Codat was awarded £5M as part of Pool D of the £425M Capability and Innovation Fund (CIF).
The conditions for accepting this award were outlining a set of public commitments to expand our offering and increase its value for UK SMEs and the financial service providers (FSPs) serving these companies.
Receiving this award, alongside recent comments from the Governor of the Bank of England, Mark Carney, are a validation of our messaging on the importance of using ‘Open Accounting’ data for SME Lending and as more lenders make the decision to start using this data, they must decide how to retrieve this data for their underwriters and credit decisioning models — do they build the integrations themselves or do they purchase a solution that does this for them?
One of our commitments to the CIF is to build 5 new integrations over the next 18 months and we felt this would be a good time to review and share our unique experience of building accounting integrations specifically for SME Lending.
I’ve broken down the article into three sections:
1. Deciding which accounting platforms to integrate with
2. Understanding the key elements of building accounting integrations
3. Building the integrations, understanding timelines and devoting technical resource
1) Deciding which accounting platforms to integrate with
The first thing to do is to identify the most popular accounting systems in your market in order to best serve your customer-base as well as set your addressable market as wide as possible. An easy way to find this information is simply asking your customers with an email survey or your relationship/account managers collecting and documenting this information over time.
From our own experience, the most popular accounting systems among SMEs in our largest target markets are:
UK & Ireland: Sage, Xero, Quickbooks (Online and Desktop), FreeAgent, Kashflow
North America: Quickbooks (Online and Desktop), FreshBooks, Oracle Netsuite, Xero, Microsoft Dynamics
Australia & New Zealand: Xero, MYOB Essentials, MYOB AccountRight, Quickbooks, Reckon
Hong Kong & Singapore: Xero, Sage, Quickbooks, MYOB
2) Understanding the key elements of building accounting integrations
Once you have decided on the most important accounting platforms for your offering, you can begin to map out the integration. We believe there are five key elements that need to be considered when designing, building and maintaining integrations across multiple accounting systems:
The Authorisation piece is key to your customer-experience and onboarding processes. Sharing data via APIs has empowered businesses to explicitly consent to sharing an agreed set of datatypes with an FSP for the purpose of their relationship and the authorisation flows designed by the accounting platforms reflect this. This is also in contrast to a screen-scraping approach, where credentials are shared with a third-party who are granted access to accounts for the purpose of creating a copy of the data.
For each accounting system’s API, the authorisation mechanism will be slightly different. Cloud solutions, for example, will be using their own flavour of common authorisation mechanisms such as OAuth, OAuth2 or sometimes a hybrid between the two.
On-premise/offline solutions such as Quickbooks Desktop or Sage50 require a direct connection between your system and the borrower’s local data file. This requires a desktop application to be built and installed on the borrower’s machine.
This is an entirely different flow, a direct connection, which requires a proprietary connector to be built and installed for each offline accounting platform, and can then be distributed, uniformly, to a large customer-base.
Standardisation is absolutely essential if you are looking to integrate with more than one accounting platform. w This isn’t only a system that retrieves data into a uniform structure, and addresses edge-cases, but also consolidates development and configuration across multiple platforms into single actions.
i) Data Standardisation:
Unlike banking data, which usually comprises of only one datatype (the bankfeed) and 5–7 fields (money in, money out, date, description, reference, balance, available balance), accounting platforms are made up of over a dozen rich datatypes with thousands of fields within each type and each accounting platform has its own nuances and structures that are used to display the data to their customers. A standardisation layer is used to address these nuances, which include different data structures, differences in language due to regions (e.g. American MMDDYY vs. UK DDMMYY date format) or different terms used for bookkeeping (e.g. Revenue vs Turnover) among others.
From the lender’s perspective, having this standardisation layer ensures the data is retrieved into your system within a common data structure and doesn’t have to be organised for processing, either to be ingested into credit models or existing underwriter and credit decisioning workflows.
A more sophisticated standardisation layer will look to incorporate patterns of customer usage by manually identifying bugs that are caused when certain edge-cases arise within your customers’ methods of bookkeeping (e.g. a customer using negative credit notes). Because of this, the more businesses you have using your API, the stronger this standardisation layer will become.
ii) Standardisation for on-premise solutions:
Unlike (often) clearly documented cloud-based APIs, desktop solutions require a deeper understanding of the layout of the underlying data which necessitates an intricate knowledge of accounting and bookkeeping, understanding the relationship between data in the underlying database tables and a longer process of investigation and trial & error before releasing the integration to your customers.
iii) API Standardisation:
Building the integrations yourselves will require you to handle development to each integration in a separate process, however, using an integration platform will instead allow your developers to leverage a standardisation layer that removes development work being duplicated. This means your developers can build your product to a single API, rather than to each accounting platform’s API individually, and will not require that you stay on top of the latest updates to each accounting platform’s API (an example of Xero’s release notes gives you an idea of how frequently this happens).
Importantly, having this standardisation layer also creates a developer experience that feels familiar and is more enjoyable and productive to build against.
iv) Standardisation for Configuration:
Using a platform means the entire integration ecosystem can be standardised, not just the data. Universal changes can be made to your preferences on synchronisation, authorisation, on-going connectivity and alerting rules, and rolled out across all of your integrations with a single action.
Each accounting system has its own rules about ongoing connectivity with its users, which is usually managed by access tokens. In an ideal world you want your customers to always be connected, so they feel comfortable that actions carried out in their accounting platform (e.g. uploading an invoice) will be synced with your platform without their further involvement but also so the data in your systems is in its most current state, which is vital for client monitoring.
Access tokens need to remain valid, otherwise you will be asking the end-user to reauthorise every time you need a refresh of the data and this will cause friction for the borrower and the lender. To do this you must be aware of the length that an authorisation token is valid for, for each accounting platform, and ensuring refreshes take place before expiration — a process which can be automated.
You will want to establish synchronisation settings so that your customers’ data is refreshed at an appropriate cadence. Syncing all your customers’ datatypes, all the time, may be attractive in an ideal world but, in reality, a segmented approach to different datatypes is necessary for navigating the challenges that need to be considered when building your integrations:
i) Accounting Platform Rate Limits:
The accounting platforms don’t appreciate when they are inundated with unnecessary API calls and, to manage the load on their systems, each accounting platforms has it’s own rules around rate limiting and load handling which must be adhered to by your product — sometimes an accounting system will even have multiple rate limits that need to be considered. When a rate limit has been breached the sync will halt and the outlying data will not be fetched.
ii) Server costs:
The more data you sync and more regularly, the higher your server costs will be. Being able to assign different cadences for each datatype, in line with your business requirements is key to containing these costs. For example, an invoice financing product may want their customers’ Purchase Ledger to sync daily, while the financial statements would only require a monthly or bi-monthly refresh in line with SME bookkeeping habits.
Using an integration platform will allow you to configure cadences for each datatype and apply them to all your accounting integrations, rather than configuring each integration individually. A platform will also take on the server cost in its entirety, as the provider will absorb the cost of high volume customers across their entire customer-base.
iii) Scaling, Peak Usage and Critical Business times:
Having the right synchronisation infrastructure in place is also vital to for when you scale, for times of peak usage and ensuring systems stay online at critical business times.
5) On-going monitoring
The final thing to be considered is in on-going monitoring of the integration, from both a user’s and a developer’s perspective.
For the end user — You’ll want an alerts system in place for your underwriters and relationship managers to react to changes in your customer’s financial data. Within the Codat platform we allow rules to be created to match particular transactional events (e.g. a new business has connected their accounting platform) or to identify trends over time (e.g. a 10% drop in sales this quarter), and we have three different notification methods to deliver this — within our portal, email or webhook.
For your developers — you will want to have a Telemetry (a project management technique) process in place that helps you monitor the functionality across your integrations. You’ll want to record the time it takes to fetch the data, monitoring up-time, monitoring error logs, data volume over time, volume of data processing for each accounting platform so you can anticipate rate limits and handle times of high usage, identifying bottlenecks in your data processing pipeline — all of which will give you the visibility you need to maintain and improve your connections.
3) Building the integrations, understanding timelines and devoting technical resource
Now you know the key components required for your integrations, you can start to factor timelines and costs for the project. Breaking this down into two sections will help — the initial build and on-going maintenance. Based on our own experience, I’ve outlined our estimates below:
The Initial Build
Required resource: One solutions architect/Product lead, two developers, 1 QA. If you’re building integrations with on-premise/offline platforms this may take longer as it will require using engineers with specific skillset, for example experience in windows desktop development
The initial build begins at the point you decide to initiate the project and will take you from the planning process through to offering an MVP:
1. Technical Architecture
Stage one is designing the technical architecture. Base this around the key elements we’ve outlined above and begin mapping the APIs from accounting platforms you have chosen to integrate with.
Estimated timeline: 1–2 months
2. Building the integrations
Estimated timeline: from our own experience, each integration has taken on average 65 days to build.
Before rolling out your integration you will want to test them internally first, and then as a ‘beta’ solution with volunteer businesses, before rolling out to your customer-base and target market.
Estimated timeline: 1 month
Based upon a project consisting of 5 integrations, you would be looking at just over a year’s development work.
Required resource: (based upon a 5 integration project) 5 engineers, 1 product manager and 1–2 customer support staff (depending on your volume of customers)
— Staying on top of the accounting platforms’ API updates will be addressed by a product manager who will then inform the engineering team to make the required changes.
— unlike banking data, which (often) comes in a uniform format, accounting data is user-enterable. Therefore addressing bugs, generated by edge use-cases within your customers’ bookkeeping, requires significant commitment from your engineering team. Each bug solved will resolve the issue for other customers who do their accounting in the same way, which is a good thing, but requires significant commitment of developer resource.
One engineer per integration required to stay on top of maintenance.
A support team is the first-line response for assisting your customers. Support staff require a certain level of knowledge on how the different accounting platforms work.
Recommended 2 support staff — one senior, one junior.
When updates take place your QA team will need to test before roll out.
The future of data retrieval in SME lending is API-based and all organisations will face the same buy vs build decision. The purpose of this article is to help lenders understand that the complexities around accounting integrations, are far more complex than summarised transaction feeds that make up an Open Banking integration.
If you’d like to know more please reach out on [email protected] or visit our API Docs