QLMSourcing Load Test Reaches Our Goal of >100000 RFQs
Well today we have >100000 RFQs in our in-memory QLMDataStore. This was the ultimate goal for the QLMSourcing application. When we started to load the system up with RFQs we hit some technical challenges and performance bottlenecks along the way. Here is the first of many blogs about our achievement.
What is a RFQ?
Lets start with what constitutes a RFQ (Request for Quote). An RFQ is a Request for Quote which consists of a Header, several line items, and is issued to several potential Vendors.
For purposes of our load test, each RFQ is filled with pseudo random data and consisting of 5 lines. Each RFQ is issued to 5 vendors. The information received from a Vendor is called a VendorQuote which has the corresponding 5 lines with the Vendor’s quote information.
For example if we wanted to send out a RFQ for bolts/washers/nuts. The RFQ would have a single header and a line for bolts, one for washers and one for nuts.
In-Memory DataStore Distribution
To achieve high speed searching of all the RFQ information in QLMSourcing, we use an in-memory database we call QLMDataStore. Due to the limitations of JVMs and operating systems we can only allocate a maximum of 1.5Gb of memory per JVM instance. This made for an interesting problem. We needed to be able to distribute the data across several JVMs and physical machines.
The solution involves using Spring RMI Invokers with an algorithm that indicates which object types are distributed and where each instance of the object should live. In the QLMSourcing case we use a modulus algorithum that splits all data across the JVMs based on mod 100 of the RFQ id. This keeps all RFQ related data together on a single JVM. One JVM is flagged as the “main” JVM for the application. This main JVM contains all non-distibuted objects such as Users, Authorization Privileges, Configuration, etc. The screen snapshot below shows the distribution of the data across 7 JVMs and 5 phsyical machines. Each remote DataStore is called “aux#”.
The QLMDataStore has some built in analysis tools to help with the distribution of the objects. The DataStore statistics page as shown below, indicates the JVM settings for each DataStore and an relative size estimate of the data being stored. The Average Excecution Time is the average time per DataStore call across the last 5000 calls. This helps us see performance bottlenecks with networks or physical machines. Note that we actually have 43 different entity object classes and >3.6 million instances of those objects. Think of it as 3.6 million rows of data in a traditional relational database.
To further analyze the data based on the type of object we display the stats per object in the DataStore. This shows how a particular object is distributed across the remote JVMs. It shows the number of instances, their relative size as compared with all objects in the system, and the indexed properties of the objects.
With over 100000 RFQs in the system, the first page of QLMSourcing filters the all the RFQ information in less than a second. This is quite fast considering the amount of information being scanned.
Look for future blog posts which will describe some of the performance issues that were addressed along the road to 100000 RFQs.