Sonic is an open source alternative solution for Elasticsearch search engine – It is designed to be lightweight, fast, and schema-less search backend. Sonic will ingest search texts and identifier tuples which you can query against in a microseconds. Sonic aims to be super-fast , crash-free, and to use minimum server hardware resources. You can see official benchmarks.

Sonic is capable of normalizing natural language search queries, auto-completing a search query and providing the most relevant results for a query. Sonic is an identifier index, rather than a document index; when queried, it returns IDs that can then be used to refer to the matched documents in an external database.

Features of Sonic Search backend

Here are some key features of Sonic search backend and why you should adopt it in your use cases.

  • Search terms are stored in collections, organized in buckets; you may use a single bucket, or a bucket per user on your platform if you need to search in separate indexes.
  • Search results return object identifiers, that can be resolved from an external database if you need to enrich the search results. This makes Sonic a simple word index, that points to identifier results. Sonic doesn’t store any direct textual data in its index, but it still holds a word graph for auto-completion and typo corrections.
  • Search query typos are corrected if there are not enough exact-match results for a given word in a search query, Sonic tries to correct the word and tries against alternate words. You’re allowed to make mistakes when searching.
  • Insert and remove items in the index; index-altering operations are light and can be committed to the server while it is running. A background tasker handles the job of consolidating the index so that the entries you have pushed or popped are quickly made available for search.
  • Auto-complete any word in real-time via the suggest operation. This helps build a snappy word suggestion feature in your end-user search interface.
  • Full Unicode compatibility on 80+ most spoken languages in the world. Sonic removes useless stop words from any text (eg. ‘the’ in English), after guessing the text language. This ensures any searched or ingested text is clean before it hits the index; see languages.
  • Simple protocol (Sonic Channel), that let you search your index, manage data ingestion (push in the index, pop from the index, flush a collection, flush a bucket, etc.) and perform administrative actions. Sonic Channel was designed to be lightweight on resources and simple to integrate with; read protocol specification.
  • Easy-to-use libraries, that let you connect to Sonic from your apps; see libraries.

Installing Sonic Elasticsearch Alternative

Sonic is coded in Rust programming language. You can install it on your system by downloading the latest release from the Sonic releases page, use cargo install or pull the source code from master.

We are going to consider two installation options for Sonic.

1. Installing Sonic from APT repository (Tested on Debian 12)

In this article we are performing an installation on Debian 12 from pre-built packages  available in an APT repository. Begin the installation with importation of GPG keys.

curl -fsSL https://packagecloud.io/valeriansaliou/sonic/gpgkey | gpg --dearmor -o /usr/share/keyrings/valeriansaliou_sonic.gpg

Add repository that contains Sonic packages after importing GPG keys.

echo "deb [signed-by=/usr/share/keyrings/valeriansaliou_sonic.gpg] https://packagecloud.io/valeriansaliou/sonic/debian/ bookworm main" > /etc/apt/sources.list.d/valeriansaliou_sonic.list

Next we install sonic package from the repository.

sudo apt update && sudo apt install sonic

Service is started automatically after the installation.

$ systemctl status sonic.service
● sonic.service - Sonic Search Index
     Loaded: loaded (/lib/systemd/system/sonic.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-03-05 16:23:21 UTC; 52s ago
   Main PID: 1534 (sonic)
      Tasks: 5 (limit: 2251)
     Memory: 9.4M
        CPU: 14ms
     CGroup: /system.slice/sonic.service
             └─1534 /usr/bin/sonic -c /etc/sonic.cfg

Mar 05 16:24:11 deb12 sonic[1534]: (DEBUG) - running a tasker tick...
Mar 05 16:24:11 deb12 sonic[1534]: (DEBUG) - scanning for kv store pool items to janitor
Mar 05 16:24:11 deb12 sonic[1534]: (INFO) - done scanning for kv store pool items to janitor, expired 0 items, now has 0 items
Mar 05 16:24:11 deb12 sonic[1534]: (DEBUG) - scanning for fst store pool items to janitor
Mar 05 16:24:11 deb12 sonic[1534]: (INFO) - done scanning for fst store pool items to janitor, expired 0 items, now has 0 items
Mar 05 16:24:11 deb12 sonic[1534]: (DEBUG) - scanning for kv store pool items to flush to disk
Mar 05 16:24:11 deb12 sonic[1534]: (INFO) - no kv store pool items need to be flushed at the moment
Mar 05 16:24:11 deb12 sonic[1534]: (DEBUG) - scanning for fst store pool items to consolidate
Mar 05 16:24:11 deb12 sonic[1534]: (INFO) - no fst store pool items to consolidate in register
Mar 05 16:24:11 deb12 sonic[1534]: (INFO) - ran tasker tick (took 0s + 0ms)

Sonic default configuration file can be edited

sudo vim /etc/sonic.cfg

Service should always be restarted after making configuration changes.

sudo systemctl restart sonic

We can test functionality by connecting to port 1491 using telnet.

$ telnet localhost 1491
Trying ::1...
Connected to localhost.
Escape character is '^]'.
CONNECTED <sonic-server v1.4.8>

2. Running Sonic in a container

Sonic can also be run in a container using container engines such as Docker or Podman. Pre-built Sonic image are available in Docker Hub as valeriansaliou/sonic.

Container method can proof to be simpler and works on multiple platforms provided Docker Engine or Podman can be installed. Refer to the following guide for steps used to install Docker Engine on Linux systems.

After Docker installation, pull the valeriansaliou/sonic image:

$ docker pull valeriansaliou/sonic
Using default tag: latest
latest: Pulling from valeriansaliou/sonic
63b450eae87c: Pull complete
960043b8858c: Pull complete
b4ca4c215f48: Pull complete
eebb06941f3e: Pull complete
02cd68c0cbf6: Pull complete
d3c894b5b2b0: Pull complete
b40161cd83fc: Pull complete
46ba3f23f1d3: Pull complete
4fa131a1b726: Pull complete
a0c7b513d518: Pull complete
fb0e36b61053: Pull complete
1a6741239af9: Pull complete
10fafdabb291: Pull complete
4031e78a5b7a: Pull complete
d5848522edae: Pull complete
d0e07d753d02: Pull complete
75e633f04497: Pull complete
Digest: sha256:9e66408d5aad76b0ad300e7a828ce46e21e9d737ad27d575508c17cce5000df8
Status: Downloaded newer image for valeriansaliou/sonic:latest
docker.io/valeriansaliou/sonic:latest

You can also pull specified version tag instead of using the latest tag. See example;

docker pull valeriansaliou/sonic:v1.4.8

Create data persistent directory for Sonic.

mkdir ~/sonic_data && cd ~/sonic_data

Create configuration file required to start Sonic service.

$ vim config.cfg
# Sonic
# Fast, lightweight and schema-less search backend
# Configuration file
# Example: https://github.com/valeriansaliou/sonic/blob/master/config.cfg


[server]

log_level = "debug"


[channel]

inet = "0.0.0.0:1491"
tcp_timeout = 300

auth_password = "SecretPassword"

[channel.search]

query_limit_default = 10
query_limit_maximum = 100
query_alternates_try = 4

suggest_limit_default = 5
suggest_limit_maximum = 20

list_limit_default = 100
list_limit_maximum = 500


[store]

[store.kv]

path = "/var/lib/sonic/store/kv/"

retain_word_objects = 1000

[store.kv.pool]

inactive_after = 1800

[store.kv.database]

flush_after = 900

compress = true
parallelism = 2
max_files = 100
max_compactions = 1
max_flushes = 1
write_buffer = 16384
write_ahead_log = true

[store.fst]

path = "/var/lib/sonic/store/fst/"

[store.fst.pool]

inactive_after = 300

[store.fst.graph]

consolidate_after = 180

max_size = 2048
max_words = 250000

Then start the container

docker run -d \
  --name sonic \
 -p 1491:1491 \
 -v ./config.cfg:/etc/sonic.cfg \
 -v ./store/:/var/lib/sonic/store/ \
 valeriansaliou/sonic

Check status to confirm if it’s running.

$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS         PORTS                                       NAMES
7c6cbfda0cbe   valeriansaliou/sonic   "sonic -c /etc/sonic…"   2 minutes ago   Up 2 minutes   0.0.0.0:1491->1491/tcp, :::1491->1491/tcp   sonic

Test connection to Sonic backend service.

$ telnet 127.0.0.1 1491
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
CONNECTED <sonic-server v1.4.8>

Using Docker Compose

Stop running container.

docker rm  -f sonic

Create compose file.

$ vim docker-compose.yml
services:
  influxdb:
    container_name: sonic
    image: valeriansaliou/sonic
    ports:
      - "1491:1491"
    volumes:
      - ./config.cfg:/etc/sonic.cfg
      - ./store/:/var/lib/sonic/store/ 
    restart: always

Start the container using docker compose command.

$ docker compose up -d
[+] Running 1/2
 ⠦ Network sonic_data_default  Created                                                                                                                                                           0.6s
 ✔ Container sonic             Started                                                                                                                                                           0.5s

Check if the container is running or if any errors are encountered.

$ docker compose ps
NAME      IMAGE                  COMMAND                  SERVICE    CREATED         STATUS         PORTS
sonic     valeriansaliou/sonic   "sonic -c /etc/sonic…"   influxdb   9 minutes ago   Up 9 minutes   0.0.0.0:1491->1491/tcp, :::1491->1491/tcp

Visit Sonic project pages for more documentation on its usage.

LEAVE A REPLY

Please enter your comment!
Please enter your name here