The enhancement of a price comparison website WeighMyRack

Goals

To cut a long story short, the ADCI Solutions developers mainly care(d) about these goals and tasks:

• Scaling of import functionality (for 10+ external data feeds) 
• Drupal updates
• Bug fixing of existing functionality
• Technical support
• Improving website performance, in particular - optimizing the work and updating data feeds.

Challenges

1. There are two servers: one serving the users and the additional sync-server processing retailers feeds. 
2. A lot of hacked contributed modules at the very beginning of our work.

 
Were these challenges dangerous? Kinda. First of all, there might have happened changes on each of the servers: for example, a content manager could apply changes on the main server while there’s an update on the sync-server. We had to figure out how to save all the newly generated data.
 
The hacked modules could get the website down if we tried to update them to the newest version. 

Solution

The servers system

We were worried whether a live website will have the newest information from retailers feeds if any server goes down. It required us an effort to make 2 servers work with a database in lockstep because this database is being constantly changed by one or another. One of the servers could go down and we wouldn’t have relevant and updated information.

To get rid of this problem, we created a custom backup-storage on a remote server. That allowed us to make sure that new information will be processed to production (the live website).

It’s worth saying that we already had the backup_migrate module set up and weekly backups provided by the Digital Ocean hosting. 

Another problem was DDoS attacks: a database MySQL resisted to work because it couldn’t process queries. As a consequence, the website was down. For this reason, we set up Monit - a system for monitoring and reanimating the processes. 

Now, if MySQL, Apache, httpd don’t respond in 3 minutes, Monit restarts the respective process/service and the website starts working again. It is extremely helpful when a DDoS attack starts in the middle of the night.

The retailers feeds

The website had 2 feeds that were updating 3-5 hours on average. As the website was scaling, and we were adding more retailers, at one point, this total import time crossed 24 hours and the whole idea of every day update become unreal taking into account the existing approach (retailers launched at a particular time, one after another). 

We decided to separate the feeds from each other and run them simultaneously. Due to the Drupal specifics, several queries tried to get access to the same information, and in the end, Drupal just overrode changes made by concurrent imports.

We implemented the lock mechanism to keep control over the data and don’t let other processes intervene until this data isn’t needed for a current process anymore.

As a result: we decreased the update time for all the feeds from 26-28 hours down to 12-15 hours. 

Results

What was improved?

1. We cleaned the code and those hacked modules. 
2. Updated all the contribs as the last security updates were 2-3 years old.
3. We refactored the import system and decreased the update time by 2 times - from 26-28 hours to 12-15 hours.
4. CX (customer experience) was improved. Now website visitors can compare prices in more stores across the world: Americas, Europe, UK, Australia, Canada.