Improving performance by removing anti-patterns and optimizing fetches
Finix has experienced tremendous growth since we entered the payments industry a few years ago. To keep pace with this growth rate, we’ve worked hard to rapidly develop new features while investing in the infrastructure needed to keep technical debt and scalability challenges in check. To pave the way for future growth, we worked on clearing out our tech debt by first working on improving performance and optimizing database fetches.
In future blog posts, we’ll describe other scalability challenges and how Finix’s Engineering team managed to navigate and overcome these issues. This blog discusses the first two challenges that we overcame to improve system performance:
Improving performance by migrating from the Hibernate framework to JOOQ, a database mapping library
Optimizing database fetches by removing anti-patterns
When engineering teams deliver new features or functionality, they will often need to refactor their code later for better performance. This is often referred to as tech debt. All software companies have some form of tech debt. A subset of tech debt are anti-patterns which are processes/code that function well in the short-term but eventually negatively impact the overall system.
Optimizing fetches by moving out of Hibernate into JOOQ
One anti-pattern we identified in our Gateway was our codebase was fetching more resources than necessary when issuing database calls while processing a transaction. Object relational mapping (ORM) frameworks, such as Hibernate, although helpful, can easily hide the true cost of a line of code. In our case, the cost was an excessive amount of fetch calls, which caused high database load. During times of high load, these excessive fetches can result in poor system performance and eventually API degradation. Below is an example of how our database worked under Hibernate.
Example: merchantRepository.findOne(UUID id)
Looking at the code example below, the reasonable expectation is the query executed against the database would look like this:
select * from merchants where id = 'xxxxx”';
Using the JProfiler tool, below is what was executed by Hibernate when we invoked the call:
As seen in the above example, one line of code equals eleven database queries. Aside from the sheer number of queries, the queries were particularly expensive as they were joined across multiple data tables. The cause of the excessive number of queries is rooted in how we implemented the code base with Hibernate annotations.
By default, Hibernate annotations (see @OneToOne and @ManyToOne) fetch types are set to “eager” mode. This setting default loads all the relationships related to a particular object.
For example, when we read from the Merchants table, we also have to read all other objects associated with it. When we traced into the objects, we noticed other associated relations would also be loaded, which put a lot of stress on our implementation. The takeaway is that we were unnecessarily doing selects on several tables:
Why we chose JOOQ
After much discussion regarding the limitations of our database and Hibernate, we made the decision to move over to JOOQ. JOOQ is a SQL centric database mapping software library. We chose JOOQ over other software because of the following reasons:
Ability to catch errors at compile time: As an engineering team, we strive to write code that proactively prevents and avoids bugs before they go into production. JOOQ generates Java classes from the database metadata. Discrepancies are alerted at compile time instead of runtime.
Simplicity in SQL: JOOQ’s framework allows engineers to know the exact SQL being run, which helps avoid surprises and reveals annotations that need to be known in order to run SQL.
Increased code visibility: All queries are statically defined in code and dynamically parameterized in JOOQ, whereas Hibernate and its object management layer dynamically generate the queries, which can differ greatly from what one might expect.
Below is an example of the improvements after implementing JOOQ:
Example: merchantJooqRepository.findOne(UUID id)
Take a look at the same code pattern after we refactored it using the JOOQ framework.
The next step in improving the performance of our code base was removing another common anti-pattern: we fetched data, used it, only to then fetch the same data later. This resulted in numerous database queries. Below is an example of the code flow for creating an Authorization (e.g. a hold on a credit card before capturing it) that would result in several database queries. We fetch the same two objects six times, resulting in twelve database calls to read the data when two fetches are sufficient.
Solving for the anti-pattern
We solved this anti-pattern by refactoring our code to use a pattern where we fetch the data that we need at the beginning of the call stack, and then pass the data down into methods. Refactoring the data resulted in the following code flow:
Controller → UnvalidatedForm → ValidatedForm → Service → Repository
The Controller is the entry point for all HTTP calls. This is where we apply our spring security permission checks.
The Unvalidated Form is responsible for validating the information provided in the API request. UnvalidatedForms provide access to a component to inject any repositories or components that may be needed to validate a request. This returns a ValidatedForm.
An example of this workflow is when an UnvalidatedForm takes in the sourceID for a Payment Instrument. The UnvalidatedForm is responsible for checking the following:
The sourceId is provided
The externalId can be parsed to internalId
The API user making the request has access to the actual source instrument
Provide the ValidatedForm the full source object
The UnvalidatedForm contains all the keys provided via API. Keys are represented by JSON and can be:
Number (Integer, Long)
Enum (variation of string)
The ValidatedForm contains all fully fetched information used to validate the request in the UnvalidatedForm. The ValidatedForm also contains additional information needed for the service. The goal is to have all the information required for the service to execute as a member variable in the ValidatedForm. This helps prevent additional database fetches.
The Service is where all the important business logic resides. The Service layer takes in the ValidatedForm; data will not be refetched. If a Service needs to call another Service, it can be done by creating and passing in that service’s ValidatedForm.
The Repository manages the inserts, updates, and reads from the database.
This is the new workflow to create an Authorization after we removed the anti-pattern:
API request comes in.
Parse request body into UnvalidatedForm.
UnvalidatedForm fetches data, validates the request, and passes the fetched objects into the ValidatedForm. At this point we can run all the validation data at the very beginning. Return early once we know the request will not be processed.
ValidatedForm now contains all the necessary data to execute the business logic.
Service layer accepts the ValidatedForm as a method argument and executes the business logic. At this point no new read database queries should be necessary and the only database calls should be inserts and updates.
The read database calls occur in step 3, thereby reducing the number of reads down from 12 to 2.
Stats show improvements
After migrating from Hibernate to JOOQ and working on removing an anti-pattern, we saw a significant reduction in the number of database calls on our system. These scalability improvements reduced contention for fetching resources, reduced API response times by ~40%, and reduced settlement file processing time by ~47%.
Thanks for reading. Stay tuned for future blog posts, where we’ll explore how Finix’s engineering team overcame other scalability challenges like optimizing monitoring and incident management, preparing test environments for scale, and upgrading our settlement engine.
If you’d like to help Finix build the most accessible financial services ecosystem in human history, starting with payments, please check out our open roles. We’re hiring engineers in San Francisco, Chicago, and remotely.
- BlogPublished 09.28.21
Stop Hacking Your Way to SaaS GrowthThe SaaS industry has been thriving for quite some time now. It’s no surprise that new companies are constantly popping up, looking for a piece of this ever-growing market. Earlier SaaS concepts targeted broader markets and are referred to as horizontal SaaS, for their wide appeal. What we’re seeing now (and for the last 10 years or so) is the verticalization of SaaS platforms. Products being built for a very specific industry with benefits directly targeted to the needs of that market. Whether a platform is built for attorneys or yoga studios, SaaS companies building for a specific vertical need to provide value to their customers on every level - building towards scale. Instead of hacking for growth, below are 3 tips for solidifying growth.
- BlogPublished 04.12.22
Everything You Need to Know About PCI ComplianceThe Payment Card Industry Data Security Standard, also known as PCI-DSS (PCI), can be a scary concept—for those new to the payments industry and payments veterans alike—because falling out of compliance with PCI standards can have serious consequences. The payments experts at Finix like to make PCI compliance less scary, by making sure our customers and community are well-informed when accepting card payments.
- BlogPublished 05.20.21
Payments Strategies for B2B SaaS PlatformsEvery Software-as-a-Service (SaaS) company needs payments. Whether by bank transfer or card payments, all SaaS companies charge their customers on a recurring basis for the software they provide. As a result, most payments advice for SaaS companies focuses on optimizing pricing and recurring billing operations to improve customer acquisition and retention ratios. While this is important work, it’s just the beginning of how SaaS companies should think about monetizing payments.