Elasticsearch Matching System

ElasticsearchSymfonyDockerMySQL

Architecture Overview

Hover over the highlighted elements in the diagram to reveal additional details.

My Profile
gulftalent.com
On-site
Server
    Symfony App
APIs
Contains:
Background jobs
Contains:
Candidate APIs
Candidates
...
...
...
Candidate Preferences
...
...
...
Candidate Experiences
...
...
...
•
•
•
Jobs
...
...
...
Job Required Skills
...
...
...
•
•
•
Job Preferences
...
...
...
ORM update listener decides whether to trigger Job matching. Conditions are: 1) The updated entity (candidate) has a minimum threshold of completeness (e.g. 70%). 2) The update includes a field that's a part of the matching algorithm.ORM update listener decides whether to trigger Job matching. Conditions are: 1) The updated entity (candidate) has a minimum threshold of completeness (e.g. 70%). 2) The update includes a field that's a part of the matching algorithm.
Trigger Job matching?
Save data
Candidate-to-Job Matching
This is to prevent multiple triggers while the candidate is updating their profile.This is to prevent multiple triggers while the candidate is updating their profile.
1) Wait for further updates if any
2) Load all candidate data
Utilize index mappings & analyzers to normalize the fields.Utilize index mappings & analyzers to normalize the fields.
3) Update/create & index the ES document
The search criteria assigns different weights to different field matches. It also sets a minimum threshold match score to only get relevant matches.The search criteria assigns different weights to different field matches. It also sets a minimum threshold match score to only get relevant matches.
4) Run the ES query to find jobs
5) Update matches in DB
The matches here are shown to the candidate & job recruiters through the website. Plus, a periodic background job runs weekly to send new matches through emails.The matches here are shown to the candidate & job recruiters through the website. Plus, a periodic background job runs weekly to send new matches through emails.The matches here are shown to the candidate & job recruiters through the website. Plus, a periodic background job runs weekly to send new matches through emails.The matches here are shown to the candidate & job recruiters through the website. Plus, a periodic background job runs weekly to send new matches through emails.The matches here are shown to the candidate & job recruiters through the website. Plus, a periodic background job runs weekly to send new matches through emails.The matches here are shown to the candidate & job recruiters through the website. Plus, a periodic background job runs weekly to send new matches through emails.The matches here are shown to the candidate & job recruiters through the website. Plus, a periodic background job runs weekly to send new matches through emails.CandidateJobMatches
ID
CandidateId
JobId
...
...
...
Indices are split into shards, distributed & replicated across nodes.Indices are split into shards, distributed & replicated across nodes.
Jobs Index
Candidates Index
Job documents
Candidate documents
YES
Job APIs
Job-to-Candidate Matching
A mirror of Candidate-to-Job matching, for job data.
Database
All jobs & candidates are indexed initially.
This diagram shows the re-indexing flow when
data changes (focusing on candidate data)
This is hosted through a managed service (Elastic Cloud)
Node A
Node B
Node C
Elasticsearch Cluster

Project Details

Architected and developed a distributed matching system using Elasticsearch to power large-scale job-candidate matching. The system ensures scalability, fault tolerance, and high-performance search by distributing indices across multiple nodes in an ES cluster.

Project Overview

  • Designed a pipeline for indexing both candidates and jobs into Elasticsearch
  • Implemented background jobs to process candidate-to-job & job-to-candidate matches
  • Ensured resilience and high availability by distributing shards across cluster nodes
  • Optimized queries for fast retrieval of relevant candidates and jobs

System Design

  • Separate indices for Jobs and Candidates, replicated across multiple nodes
  • Symmetric matching workflows: Candidate → Job & Job → Candidate
  • Initial bulk indexing plus incremental updates through background workers
  • Visualized architecture highlighting ES cluster, indexing pipeline, and matching flow

Technical Features

  • Elasticsearch cluster with sharding and replication
  • Symfony for background job orchestration
  • Efficient search queries with custom scoring logic
  • Scalable and fault-tolerant distributed setup