TipTuti

Building fast, fuzzy site search with Laravel and Typesense

Modern applications have high demands for data storage capabilities. Over the past 10 years, the rise of purpose-built data platforms has taken off with segmentation around data and analytics, transactional, related entities and graphs, as well as search and artificial intelligence. The search space alone has seen tremendous growth which is requiring vendors to push their platforms into new and emerging areas including support for vector embeddings. All of this sounds amazing and futuristic, but what if the same platform that supported AI also supported good old traditional search? And what about support for a more human search that includes things like typographical errors? I want to explore a platform by the company Typesense and how well it pairs with the web stalwart in PHP with Laravel.

For disclosure, before I begin, this article is sponsored by Typesense, the company behind "Lightning-fast Open Source Search". I'm writing a series of articles that explores how builders can make use of their search products with various languages and setups.

Walkthrough the Design

Before diving into the code and walking through how to get started with Typesense and Laravel, I want to pause and highlight exactly what I've put together. Doing this upfront will orient the content around an outcome.

From this diagram, I'm going to work through:

  1. Highlight the outline of the project
  2. Create a Todo model that will be the basis for the data
  3. Demonstrate configuring Laravel's Scout for Typesense
  4. Illustrate model synchronization from Database to Typesense
  5. Showcase building Laravel Controllers and Views to support the model

Project Build

The basis for the following paragraphs is this GitHub repository. For full disclosure, my PHP experience goes back many years and the code included needs some updates to be "production" ready, but the real things of interest are the Typesense components and how simple Laravel makes it for keeping a database and a Typesense store in sync.

Getting Started

What you'll find when cloning the project is that it's a basic and boilerplate Laravel setup that was created from running composer create-project laravel/laravel typesense-app. The interesting parts from a Typesense standpoint start to show up when I open up the file at app/Models/Todo.php

Todo Model

My Todo model is the classic case of needing something to store that has meaning but isn't so specific that it is hard to relate to or follow. What is worth pointing out as it relates to Typesense specifically is the inclusion of use Searchable which is a model trait. By including this trait, a model observer will be registered that automatically keeps the model in sync with the database and Typesense. Incredible that with that one line of code, I get all of that functionality.

The second piece of this Todo model that is pertinent to Typesense is the toSearchableArray() function. By default, Typesense will use an id field of type string as its key for that document as referenced in this article. In addition to converting the id to a string, it's recommended that timestamps be stored as Unix epochs making it an integer.

<?php
namespace App\Models;
 
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
use Laravel\Scout\Searchable;
 
class Todo extends Model
{
    use HasFactory;
    use Searchable;
 
    protected $fillable = [
        'name',
        'description'
    ];
 
    public function toSearchableArray()
    {
        return array_merge($this->toArray(),[
            'id' => (string) $this->id,
            'created_at' => $this->created_at->timestamp,
        ]);
    }
}

Configuring Scout

With a model defined and ready to be stored, I need to configure Scout so that the observer knows where to store my data. The scout.php file is found in the config directory of my project like in the image below.

The Scout configuration file is essentially a giant array with objects for different storage platforms and drivers. This makes adding in the Typesense configuration straightforward to maintain. Two separate sections of the Typesense configuration need to be addressed.

Section one is focused on client configuration. In this section, I'm setting things like API keys, host, ports, and paths. Think of it like the driver portion of the setup. From that perspective, it feels like a normal database configuration.

'client-settings' => [
    'api_key' => env('TYPESENSE_API_KEY', '<your-api-key>'),
    'nodes' => [
        [
            'host' => env('TYPESENSE_HOST', 'localhost'),
            'port' => env('TYPESENSE_PORT', '8108'),
            'path' => env('TYPESENSE_PATH', ''),
            'protocol' => env('TYPESENSE_PROTOCOL', 'http'),
        ],
    ],
    'nearest_node' => [
        'host' => env('TYPESENSE_HOST', 'localhost'),
        'port' => env('TYPESENSE_PORT', '8108'),
        'path' => env('TYPESENSE_PATH', ''),
        'protocol' => env('TYPESENSE_PROTOCOL', 'http'),
    ],
    'connection_timeout_seconds' => env('TYPESENSE_CONNECTION_TIMEOUT_SECONDS', 2),
    'healthcheck_interval_seconds' => env('TYPESENSE_HEALTHCHECK_INTERVAL_SECONDS', 30),
    'num_retries' => env('TYPESENSE_NUM_RETRIES', 3),
],
    'retry_interval_seconds' => env('TYPESENSE_RETRY_INTERVAL_SECONDS', 1),

The second section that requires addressing is more of a collection of entities. For each class that I want to be stored and synchronized in Typesense, I need to configure those mappings. For my Todo model, those mappings include each of the fields that I desire to be in the document. That includes the field names and the datatype of that field. In addition, I can configure default_sorting as well as set up indexing for specific fields that I want to search through search-parameters.

'model-settings' => [
    Todo::class => [
        'collection-schema' => [
            'fields' => [
                [
                    'name' => 'id',
                    'type' => 'string',
                ],
                [
                    'name' => 'name',
                    'type' => 'string',
                ],
                [
                    'name' => 'description',
                    'type' => 'string',
                ],
                [
                    'name' => 'created_at',
                    'type' => 'int64',
                ],
            ],
            'default_sorting_field' => 'created_at',
        ],
        'search-parameters' => [
            'query_by' => 'name'
        ],
    ],
],

Views and Controllers

Now that I have the Todo model configured for the observer and Scout has been adjusted to map the documents, it's time to bring it all together with Laravel Views and Controllers.

In my web.php route definitions, I'm establishing the following endpoints.

  • /todos returns the list of Todos from the Typesense
  • /todos/new returns that View with the New Todo form
  • /todos/save writes the new Todo from the form into the database
  • /todos/search executes the query against Typesense
Route::get("/todos", [TodoController::class, 'index']);
Route::get('/todos/new', [TodoController::class, 'newTodo']);
Route::post('/todos/save', [TodoController::class, 'store']);
Route::post('/todos/search', [TodoController::class, 'search']);

Todos List

In my Todo list, I could fetch the data from the SQL database, but I'm opting to fetch from Typesense instead. I'd rather return data from the search database so that I have consistent data and behavior when working with the grid. The code inside of the index handler on the controller performs the query and then forms the data into a Todo model.

$array = Todo::search('')->get()->toArray();
$todos = [];
 
foreach ($array as $todo) {
    $t = new Todo;
    $t->id = $todo['id'];
    $t->name = $todo['name'];
    $t->description = $todo['description'];
    $t->created_at = $todo['created_at'];
    $t->updated_at = $todo['updated_at'];
    array_push($todos, $t);
}
 
return view('todo')->with( ['todos' => $todos] );;

Visually that renders out in my grid with the columns matching the fields on the Todo.

Creating a New Todo

The link at the top of my grid will take me to the new Todo form. As my model outlines, I've got a name and description so ultra simple in terms of what's required in the table. Working with Laravel is so nice and clean that my model includes the save() method for dealing with the database.

$todo = new Todo;
$todo->name = $request->name;
$todo->description = $request->description;
$todo->save();
 
return redirect('/todos')->with('status', 'Todo Data Has Been inserted');

Searching the Todos

With my newly created Todo I can then run my search via the input field that is at the top of my grid. Just like in the index, I'm going to execute a search, but this time, I'll be using values from the form input.

What I really appreciate about using Scout and Typesense is that as a developer, it's mostly abstracted away from me. I can focus on what my user experience is without having to worry about some of the lower-level details.

public function search(Request $request): View
{
    $search = '';
    if ($request->search) {
        $search = $request->search;
    }
 
    $array = Todo::search($search)->get()->toArray();
    $searched = [];
 
    foreach ($array as $todo) {
        $t = new Todo;
        $t->id = $todo['id'];
        $t->name = $todo['name'];
        $t->description = $todo['description'];
        $t->created_at = $todo['created_at'];
        $t->updated_at = $todo['updated_at'];
        array_push($searched, $t);
    }
 
    return view('todo')->with( ['todos' => $searched ]);
}

Wrapping Up

With the views and controllers built, I now have a working solution that performs the following features.

  • Lists the Todos in Typesense
  • Allows for searching of Typesense by way of a query from the form field
  • An HTML form creates a new Todo item
  • A Laravel controller persists the data in SQLite
  • Laravel automatically syncs my newly saved data with Typesense

By using the Laravel Framework to integrate with Typesense, I get big functionality with more configuration than the actual code written. As I add more models to my project, I have repeatable and scalable patterns for including those in my search should I desire.

Impressions and Takeaways

Searching is something that all users ask for in some capacity but it is often delivered as a series of like '%<string>%' type statements against columns in a database. This might be useful in some instances but puts a heavy burden on the developer to cover all of the scenarios and can also be very taxing on the database to keep up with the demand. It also forces the developer to get involved with database administrators to set up things like full-text indexes and support for those isn't universal.

This is where Typesense and Laravel shine. Typesense is a purpose-built search database that handles some nice default cases super well out of the box. Things like typographical mistakes are accounted for. Document building and indexing are managed through simple configurations. Then when paired with Laravel and Scout, the synchronization makes using this platform a breeze.

And this is just the beginning.

Thanks for reading and happy building!

Comments

Back