In future blog posts, we’ll describe other scalability challenges and how Finix’s Engineering team managed to navigate and overcome these issues. This blog discusses the first two challenges that we overcame to improve system performance:

1. Improving performance by migrating from the Hibernate framework to JOOQ, a database mapping library

2. Optimizing database fetches by removing anti-patterns

When engineering teams deliver new features or functionality, they will often need to refactor their code later for better performance. This is often referred to as tech debt. All software companies have some form of tech debt. A subset of tech debt are anti-patterns which are processes/code that function well in the short-term but eventually negatively impact the overall system. 

Optimizing fetches by moving out of Hibernate into JOOQ 

One anti-pattern we identified in our Gateway was our codebase was fetching more resources than necessary when issuing database calls while processing a transaction. Object relational mapping (ORM) frameworks, such as Hibernate, although helpful, can easily hide the true cost of a line of code. In our case, the cost was an excessive amount of fetch calls, which caused high database load. During times of high load, these excessive fetches can result in poor system performance and eventually API degradation. Below is an example of how our database worked under Hibernate. 

Example:  merchantRepository.findOne(UUID id) 

Looking at the code example below, the reasonable expectation is the query executed against the database would look like this: 

select * from merchants where id = 'xxxxx”';

Using the JProfiler tool, below is what was executed by Hibernate when we invoked the call:

As seen in the above example, one line of code equals eleven database queries. Aside from the sheer number of queries, the queries were particularly expensive as they were joined across multiple data tables. The cause of the excessive number of queries is rooted in how we implemented the code base with Hibernate annotations. 

By default, Hibernate annotations (see @OneToOne and @ManyToOne) fetch types are set to “eager” mode. This setting default loads all the relationships related to a particular object.

For example, when we read from the Merchants table, we also have to read all other objects associated with it. When we traced into the objects, we noticed other associated relations would also be loaded, which put a lot of stress on our implementation. The takeaway is that we were unnecessarily doing selects on several tables:

Why we chose JOOQ

After much discussion regarding the limitations of our database and Hibernate, we made the decision to move over to JOOQ. JOOQ is a SQL centric database mapping software library. We chose JOOQ over other software because of the following reasons: 

• Ability to catch errors at compile time: As an engineering team, we strive to write code that proactively prevents and avoids bugs before they go into production. JOOQ generates Java classes from the database metadata. Discrepancies are alerted at compile time instead of runtime. 

• Simplicity in SQL: JOOQ’s framework allows engineers to know the exact SQL being run, which helps avoid surprises and reveals annotations that need to be known in order to run SQL. 

• Increased code visibility: All queries are statically defined in code and dynamically parameterized in JOOQ, whereas Hibernate and its object management layer dynamically generate the queries, which can differ greatly from what one might expect.

Below is an example of the improvements after implementing JOOQ: 

Example: merchantJooqRepository.findOne(UUID id)

Take a look at the same code pattern after we refactored it using the JOOQ framework.

The next step in improving the performance of our code base was removing another common anti-pattern: we fetched data, used it, only to then fetch the same data later. This resulted in numerous database queries. Below is an example of the code flow for creating an Authorization (e.g. a hold on a credit card before capturing it) that would result in several database queries. We fetch the same two objects six times, resulting in twelve database calls to read the data when two fetches are sufficient.

We solved this anti-pattern by refactoring our code to use a pattern where we fetch the data that we need at the beginning of the call stack, and then pass the data down into methods. Refactoring the data resulted in the following code flow: 

Controller → UnvalidatedForm → ValidatedForm  → Service →  Repository

Controller

The Controller is the entry point for all HTTP calls. This is where we apply our spring security permission checks. 

UnvalidatedForm 

The Unvalidated Form is responsible for validating the information provided in the API request. UnvalidatedForms provide access to a component to inject any repositories or components that may be needed to validate a request. This returns a ValidatedForm. 

An example of this workflow is when an UnvalidatedForm takes in the sourceID for a Payment Instrument. The UnvalidatedForm is responsible for checking the following: 

• The sourceId is provided

• The externalId can be parsed to internalId

• The API user making the request has access to the actual source instrument

• Provide the ValidatedForm the full source object

The UnvalidatedForm contains all the keys provided via API. Keys are represented by JSON and can be: 

• String

• Number (Integer, Long)

• Enum (variation of string)

• Array

• Object

• Boolean

ValidatedForm 

The ValidatedForm contains all fully fetched information used to validate the request in the UnvalidatedForm. The ValidatedForm also contains additional information needed for the service. The goal is to have all the information required for the service to execute as a member variable in the ValidatedForm. This helps prevent additional database fetches. 

Service 

The Service is where all the important business logic resides. The Service layer takes in the ValidatedForm; data will not be refetched. If a Service needs to call another Service, it can be done by creating and passing in that service’s ValidatedForm.  

Repository

The Repository manages the inserts, updates, and reads from the database.

This is the new workflow to create an Authorization after we removed the anti-pattern: 

1. API request comes in.

2. Parse request body into UnvalidatedForm.

3. UnvalidatedForm fetches data, validates the request, and passes the fetched objects into the ValidatedForm. At this point we can run all the validation data at the very beginning. Return early once we know the request will not be processed.

4. ValidatedForm now contains all the necessary data to execute the business logic.

5. Service layer accepts the ValidatedForm as a method argument and executes the business logic. At this point no new read database queries should be necessary and the only database calls should be inserts and updates.

The read database calls occur in step 3, thereby reducing the number of reads down from 12 to 2.

After migrating from Hibernate to JOOQ and working on removing an anti-pattern, we saw a significant reduction in the number of database calls on our system. These scalability improvements reduced contention for fetching resources, reduced API response times by ~40%, and reduced settlement file processing time by ~47%. 

Thanks for reading. Stay tuned for future blog posts, where we’ll explore how Finix’s engineering team overcame other scalability challenges like optimizing monitoring and incident management, preparing test environments for scale, and upgrading our settlement engine. 

If you’d like to help Finix build the most accessible financial services ecosystem in human history, starting with payments, please check out our open roles. We’re hiring engineers in San Francisco, Chicago, and remotely.