Public API for Search

Business logic

The main purpose provide search by stages, teams, actors and categories.

Updated internal models get from listening kafka topics: search-api-participants, search-api-stages, search-api-popular, search-api-regions, search-api-categories, search-api-actors.

The main advantage is to load and store a full offer in memory.

The client response is created only from in-memory data, it allow to avoid slow database queries and minimises response times.

Has dependency from com.livescore.public-service-sdk: https://github.com/minelytix/spring-common-processor

OpenAPI specification

Swagger UI

Internal Models

Handle internal models: participant, stage, season, region, category, actors. Internal models retrieved from kafka topics are written to mongodb collections and Internal cache.

Kafka

Listening topics: search-api-participants, search-api-stage, search-api-popular, search-api-regions, search-api-categories, search-api-actors.

Mongodb

Write resulting models into collections: participants, stages, regions, categories, seasons, popular-ids, actors.

Startup

On startup all data from mongodb collections (participants, stages, regions, categories, season, popular-ids, actors) is loaded into an internal cache. After loading application is active and mongo connection closes.

Internal cache

Loaded from DB and stored in IndexedCollections. Used technology - CQ Engine https://github.com/npgall/cqengine

CDN cache

Not customized

VM instance configuration recommendations

CPU: 2 cores, RAM: 2 Gb, HDD: 20 Gb

Metrics

Grafana dashboard: 'Search prometheus monitoring'.

There are 3 types of metrics: GAUGE, COUNTER, TIMER (contains 2 dimensions: count and sum)

Custom metrics

model_updates_total (COUNTER)

total count of handled messages, received from kafka. Tags: model, operation

Predefined metrics

jetty.*: metrics show the state of jetty thread pool.

http_server_requests_seconds.* : displays how the api is used. Tags: exception, method, outcome, status, uri

spring_kafka_listener_seconds.* : displays information on kafka connection and consumption. Tags: exception, name, result

Test REST API Endpoints

Not present

LeagueSearch

Endpoint do stage search by query or return popular stages from cache.

Request example

GET /v1/api/react/stage/soccer/search HTTP/1.1
Content-Type: application/json;charset=UTF-8
Content-Length: 50
Host: example.com:8087

Response example

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: max-age=100, public
x-response-time: 2 ms
Content-Length: 757

{
    "Stages": [
        {
            "Snm": "Premier League",
            "Sds": "Premier League",
            "Scd": "premier-league",
            "Sid": "7126",
            "Cnm": "England",
            "Csnm": "England",
            "Ccd": "england",
            "Cid": "34",
            "Scu": 0,
            "Chi": 0,
            "Shi": 0,
            "Ccdiso": "GBR",
            "CompId": "10",
            "CompN": "Premier League",
            "CompD": "England",
            "CompST": "England",
            "Spid": 1,
            "suggestions":{
                "Teams":[],
            }
        },
        {
            "Snm": "Qualification",
            "Sds": "Qualification",
            "Scd": "qualification",
            "Sid": "7119",
            "Cnm": "Champions League",
            "Csnm": "Champions League",
            "Ccd": "champions-league",
            "Cid": "130",
            "Scu": 1,
            "Chi": 0,
            "Shi": 0,
            "Ccdiso": "NON",
            "CompId": "29",
            "CompN": "Champions League",
            "CompD": "UEFA",
            "Spid": 1,
            "suggestions":{
                "Teams":[],
            }
        }
    ]
}

Load testing scenarios

Scaling policy recommendations

If the processor load is more than 70%, then we launch the second node