Search, recommendation systems, retrieval-augmented generation (RAG), semantic similarity — almost every modern ML system quietly depends on fast nearest-neighbor search. And yet, for most of us, vector databases remain a black box. We use them, we tune a few parameters, but rarely stop to ask why they work the way they do — or when they stop working.

This project is my attempt to change that.

Instead of starting from a mature system or a clever approximation technique, I decided to rebuild a vector database from first principles in C++. The goal was not to compete with production-grade systems, but to understand the fundamentals: why certain designs work, where they break down, and what trade-offs they make between speed, accuracy, and complexity.

I start with the simplest possible baseline — an exact linear scan — and gradually layer in optimizations, SIMD instructions, multithreading, and tree-based indexing.

This blog documents that journey. It’s not a tutorial on “how to build the fastest vector database,” but a practical exploration of how vector search behaves on real hardware and why modern systems are designed the way they are.

If you want to dig deeper, experiment, or go through the full implementation, you can find the complete codebase here:
👉 github.com/SriPrarabdha/vector_db_from_scratch

Baseline Implementation (Linear Scan)

Every optimization only makes sense if we clearly understand what we are optimizing against.
So before touching SIMD, threads, or tree structures, the project starts with the most boring solution possible: linear scan.

Given a query vector, we compute its distance to every single vector in the database and then pick the top-k closest ones. That’s it. No shortcuts, no pruning, no approximation.

At first glance, this feels obviously inefficient. But linear scan has one extremely important property: it is exact. Whatever result it gives is the ground truth. Every faster method we build later will be judged relative to this.

How linear scan is actually working here

Assume we have N vectors, each of dimension D. For a single query:

1. Take one stored vector.

2. Compute the distance to the query (L2 / Cosine distance).

3. Repeat for all N vectors.

4. Sort (or partially select) the closest k.

The time complexity is simple and brutal: O(N × D) per query.

//square of euclidean distance
float l2_distance (const Vector& a, const Vector& b){
    assert (a.dim == b.dim);

    float sum = 0.0f;
    for(dim_t i = 0; i<a.dim ; ++i){
        float d = (a.data[i] - b.data[i]);

        sum += d*d;
    }

    return sum;
}

float cosine_distance(const Vector& a, const Vector& b){
    assert (a.dim == b.dim);

    float dot = 0.0f;
    float na = 0.0f;
    float nb = 0.0f;

    for(dim_t i = 0 ; i<a.dim ; ++i){
        dot += a.data[i] * b.data[i];
        na += a.data[i] * a.data[i];
        nb += b.data[i] * b.data[i];
    }

    if (na == 0.0f || nb == 0.0f) return 1.0f;

    return 1.0f - (dot / (sqrt(na) * sqrt(nb)));
}

Now let’s use this vanilla implementation of l2_distance to implement our linear scan index. Refer here to see the complete code implementing the Linear Scan Index class and other utility methods. Below is just the logic for the Search methods of the Linear Index Class.

// search in linear scan index
vector<pair<uint32_t , float>> LinearScanIndex::search(const Vector& query , size_t k) const {
    assert (query.dim == dim_);

    vector<pair<uint32_t, float>> results;

    results.reserve(data_.size());

    for(size_t_t i = 0; i<data_.size() ; ++i){
        dist_t d = l2_distance(query , data_[i]);
        results.emplace_back(i, d);
    }

    if(results.size() > k){
        nth_element(results.begin() , 
                    results.begin() + k,
                    results.end() , 
                    [](auto& a , auto& b) {return a.second < b.second ;}
                );
        results.resize(k);
    }

    sort(results.begin() , results.end() , 
        [](auto& a , auto& b) {return a.second < b.second ;}
    );

    return results;
}

Let’s try & Optimize this Linear Scan

At this point the algorithm is still exactly the same. We are still comparing the query against every vector. We are still computing the exact L2 distance.

The only thing that changes now is how efficiently the CPU is allowed to do the work.

Doing more work, per CPU cycle with AVX2

A modern CPU is not meant to process one floating-point number at a time. It has wide vector registers that can hold multiple values and operate on all of them in a single instruction. AVX2 gives us 256-bit registers, which means 8 floats at once.

The idea behind AVX2 distance computation is simple:
instead of computing : (a_i - b_i)²

one dimension at a time, we compute eight of them together.

If you look at the l2_avx2 routine, the core loop moves through the vectors in chunks of 8. _mm256_loadu_ps loads 8 floats from memory into a single register. One register holds part of vector a, another holds the corresponding part of vector b.

Once both are in registers, subtraction happens in parallel for all 8 values. Squaring is also done in parallel, and the results are accumulated into a running sum. This accumulation uses a fused multiply-add instruction, which computes : sum = diff² + sum

in one step. This is both faster and numerically stable. What is important here is that the inner loop becomes branch-free and dense. The CPU sees a tight stream of predictable instructions, which is exactly what it likes.

float l2_avx2(const float* a, const float* b, size_t dim) {
    __m256 sum = _mm256_setzero_ps();
    size_t i = 0;

    for (; i + 8 <= dim; i += 8) {
        __m256 va = _mm256_loadu_ps(a + i);
        __m256 vb = _mm256_loadu_ps(b + i);
        __m256 diff = _mm256_sub_ps(va, vb);
        sum = _mm256_fmadd_ps(diff, diff, sum);
    }

    float tmp[8];
    _mm256_storeu_ps(tmp, sum);

    float res = tmp[0] + tmp[1] + tmp[2] + tmp[3]
              + tmp[4] + tmp[5] + tmp[6] + tmp[7];

    for (; i < dim; ++i) {
        float d = a[i] - b[i];
        res += d * d;
    }

    return res;
}

Important Thing to Note : AVX does not do automatically is reduce a vector register to a single scalar. After the loop finishes, the sum still lives in a register holding 8 partial values.

To finish the distance computation, the register is stored back to memory and the 8 values are added together in normal scalar code. This step looks a bit ugly, but its cost is negligible compared to the main loop, especially for high-dimensional vectors.

Any leftover dimensions that do not fit into a multiple of 8 are handled at the end using a simple scalar loop. This ensures correctness for all dimensions without complicating the main vectorized loop. AVX2 does not change the number of arithmetic operations mathematically. It changes how many are executed per CPU cycle.

Instead of issuing one subtract and one multiply per dimension, the CPU now issues eight of them at once. Memory access also becomes more efficient because data is consumed in wide, predictable chunks.

Below table is for the following setting : Dataset size : 100000 , Dimension : 1024 , Queries : 100 ,Top-K : 10

Multi-threading with OpenMP

Once distance computation is fast, the next bottleneck becomes obvious: there are still many vectors to process.

Each query–vector distance is independent of every other one. So our mind very quickly asks the question can we use Multi-Threading . The easiest form of parallelism is using OpenMP at the outer level which allows the loop over database vectors to be split across CPU cores with almost no code changes.

The key lesson here is that multi-threading only helps after the inner loop is efficient. Parallelizing slow code just gives you slow code running on multiple cores.

#include <omp.h>

vector<pair<idx_t , dist_t>> LinearScanIndex::search(const Vector& query , size_t k) const {
    assert (query.dim == dim_);

    vector<pair<idx_t, dist_t>> results(aos_.size());

    auto compute = [&] (idx_t i) {
        dist_t d = l2_dispatch(query , aos_[i] , cfg_.distance);
        results[i] = {i , d};
    };

    if(cfg_.exec == ExecPolicy::OPENMP){
        #pragma omp parralel for schedule(static)
        for(idx_t i = 0 ; i<aos_.size() ; i++)compute(i);
    }else{
        for(idx_t i = 0 ; i<aos_.size() ; i++) compute(i);
    }

    if(results.size() > k){
        nth_element(results.begin() , 
                    results.begin() + k,
                    results.end() , 
                    [](auto& a , auto& b) {return a.second < b.second ;}
                );
        results.resize(k);
    }

    sort(results.begin() , results.end() , 
        [](auto& a , auto& b) {return a.second < b.second ;}
    );

    return results;
}

Below table is for the following setting : Dataset size : 100000 , Dimension : 1024 , Queries : 100 ,Top-K : 10 , threads : 12

Memory layout: AoS vs SoA

Another optimization that matters more than expected is how vectors are stored in memory. With an array-of-structures layout, each vector is stored contiguously, but different vectors are interleaved. With a structure-of-arrays layout, each dimension is stored contiguously across all vectors.

For linear scan with AVX2, contiguous memory access is critical. When data is laid out in a cache-friendly way, the CPU can prefetch aggressively and avoid stalling on memory. A bad layout can completely erase the gains from SIMD and multithreading.

This is where theory meets reality: the same algorithm, the same math, but very different performance depending on layout.

KD Tree

After squeezing everything we could out of linear scan, the next logical step is to reduce the amount of work altogether. Instead of comparing the query against every vector, the goal now is to avoid computing distances that clearly cannot matter.

This is where KD-Trees come in. A KD-Tree recursively partitions the vector space by splitting along one dimension at a time. At each node, vectors are divided based on whether they lie on one side of a splitting value or the other. Over time, this creates a tree where nearby points in space tend to live close together in the structure.

The key idea is that if a region of space is far from the query, we can skip all vectors inside it without ever computing their distances.

Building The Tree

The KD-Tree is constructed top-down. At the root, all vectors are considered. A splitting axis is chosen based on the current depth, usually by cycling through dimensions. Along that axis, vectors are sorted and split at the median. This guarantees that the tree remains roughly balanced.

Each node stores the splitting axis, the split value, and pointers to its left and right children. Leaf nodes contain a small set of vectors and act as stopping points for recursion.

This construction cost is paid once, offline. The real question is whether it pays off during search.

Querying the Tree

Searching the KD-Tree starts by descending into the subtree that contains the query point. This is the “greedy” path, following the side of the split where the query lies.

Once a leaf is reached, distances to its vectors are computed and the current best neighbors are updated. Then comes the crucial part: backtracking.

When backtracking, the algorithm checks whether the other side of a split could possibly contain a closer neighbor than the current best. This decision is made using geometry. If the distance from the query to the splitting plane is larger than the worst distance in the current top-k set, the entire subtree can be safely pruned.

This pruning is what makes KD-Trees fast in low dimensions. Large parts of the dataset are never touched.

The curse of Dimensionality Shows up

As dimensionality increases, something interesting — and slightly depressing — happens.

Distances become less informative. Points start to look equally far from the query in all directions. The distance to the splitting plane rarely exceeds the current best distance, which means pruning almost never triggers.

In high dimensions, the KD-Tree ends up visiting most of the nodes. At that point, the search degenerates back into something very close to linear scan, with extra overhead from recursion and tree traversal.

Below table is for the following setting : Dataset size : 100000 , Dimension , Queries : 1 ,Top-K : 10

This is not a bug. It is a fundamental limitation . Implementing KD-Trees makes this painfully clear in practice, not just in theory.

LangGraph is built on top of LangChain. One of the common patterns we see when people are creating more complex LLM applications is the introduction of cycles into the runtime. These cycles often use the LLM to reason about what to do next in the cycle. A big unlock of LLMs is the ability to use them for these reasoning tasks. This can essentially be thought of as running an LLM in a for-loop. These types of systems are often called agents.

It adds new value primarily through the introduction of an easy way to create cyclical graphs. This is often useful when creating agent runtimes.

Don't worry if this all seems very daunting (it was the same for me😨 not a long time ago). We will break down how this state-of-the-art library works and how we can use it to make amazing AI Agents.

LagGraph: Level 1

Basic Components in the LangGraph Workflow:

1. Node - Think these node are part of a graph and they just call a function on the input they have and pass the output. Nothing else

2. Edge - Like any graph it is the connection between any 2 node where output of one node is passed as input to the second node

3. Entry_Point - It's just a node which gets the initial user prompt as it's input

4. Finish_Point - It's also just a node that gives the output to the user

The above illustration is an example of the simplest graph where we can see node 1,2 are just calling function 1,2. And the output of node 1 or function call 1 is a input for the node 2/ function call 2

Define Nodes (aka functions) --> Define edges between Nodes --> Define Entry Point --> Define Finish Point

Also node 1 acts as a starting point as it gets the initial user prompt as input. And node 2 is the Finish_Point as it's output is the Graph Output to the user.

Now let's try to code this out step by step. First, let's install all the dependencies for this project

pip install langchain-openai langgraph python-dotenv

def function1(input1:str)->str:
    return input1 + " Hi"

def function2(input2:str) -> str:
    return input2+ " There"

The above code snippet defines 2 very basic functions. 'Function1' just adds "Hi" to it's input and returns it. And 'Function2' just adds "There" to its input and returns it.

Now the JUICY Part:

from langgraph.graph import Graph

Workflow = Graph()

Workflow.add_node("node_1", function1)
Workflow.add_node("node_2", function2)

Workflow.add_edge('node_1', 'node_2')

Workflow.set_entry_point("node_1")
Workflow.set_finish_point("node_2")

app = Workflow.compile()

Just like discussed before we set up 'function1' as "node1" and 'function2' as "node2". Add an edge between the 2 nodes. Set "node1" as entry point and "node2" as the finish point. And that's it. This is the main ideology behind LangGraph. And now we can use this exact same method to build the strongest AI Agent.

Now run this agent on the given user prompt

app.invoke("hello")

'hello Hi There'

And that's all it takes to build with LangGraph😎

LangGraph: Level 2

Now that we have seen the basics of LangGraph let's make our Workflow a little more useful by adding LLM call

This is a simple code snippet using Langchain to use chat models of openai

from langchain_openai import ChatOpenAI

model = ChatOpenAI(temperature=0.8)

model.invoke("Hey there").content

'Hello! How can I assist you today?'

Now we just have to modify our functions to handle calling of a llm model

def func1(input1: str)->str:
    return model.invoke(input1).content

def func2(input2: str)->str:
    return "Agents says:" + input2

Now just follow the previous Workflow

Define Nodes (aka functions) --> Define edges between Nodes --> Define Entry Point --> Define Finish Point

from langgraph.graph import Graph

Workflow = Graph()

Workflow.add_node("node_1", func1)
Workflow.add_node("node_2", func2)

Workflow.add_edge('node_1', 'node_2')

Workflow.set_entry_point("node_1")
Workflow.set_finish_point("node_2")

app = Workflow.compile()

And now with everything set up run our LangGraph Agent

app.invoke("aaj date kya h?")

'Agents says: Maine Internet ki permission nahi hai, isliye main aapko current date and time provide nahi kar sakti. Aap apne device ya kisi aur source se current date aur time pata kar sakte hain.'

LangGraph: Level 3

Now that we have added LLM function calling to our LangGraph Agent there is just one thing left to make an amazing AI Agent i.e. Tools

Tools gives us extraordinary power by providing some extra information that the agent needs and solve any complex problem. Here we will be using 'OpenWeatherMap' API to get the weather of any city using our Agent

Visit this link to make an account with Open Weather Map org and you will get your API Key without the need to use your Credit Card. Now let's go through the code to use its Langchain wrapper and the weather report of any city

pip install pyowm

from langchain_community.utilities import OpenWeatherMapAPIWrapper
from dotenv import load_dotenv
import os

load_dotenv()

os.environ["OPENWEATHERMAP_API_KEY"] = os.environ.get("OPENWEATHERMAP_API_KEY")

weather = OpenWeatherMapAPIWrapper()

weather_data = weather.run("New Delhi")
print(weather_data)

In New Delhi, the current weather is as follows:
Detailed status: haze
Wind speed: 2.57 m/s, direction: 300°
Humidity: 18%
Temperature: 
  - Current: 33.09°C
  - High: 33.09°C
  - Low: 33.09°C
  - Feels like: 30.9°C
Rain: {}
Heat index: None
Cloud cover: 0%

Now we have to modify our 'function1' in such a way that given any prompt the LLM call output just contains the name of the city . And 'function2' just picks the name of city generated before and uses OpenWeatherMapAPIWrapper() to get the weather report for that city

def function1(prompt: str)->str:
    complete_prompt = f"Your task is to provide only the city name based on the user query. Nothing more, just the city name mentioned. Following is the user query: {prompt}"
    return model.invoke(complete_prompt).content

def function2(input1: str)->str:
    return weather.run(input1)

Now let's compile our functions into the nodes of our agents and invoke our LangGraph Agent

from langgraph.graph import Graph

Workflow = Graph()

Workflow.add_node("agent", function1)
Workflow.add_node("tool", function2)

Workflow.add_edge('agent', 'tool')

Workflow.set_entry_point("agent")
Workflow.set_finish_point("tool")

app = Workflow.compile()
app.invoke("What is the weather right now in Kanpur")

'In Kanpur, the current weather is as follows:\nDetailed status: clear sky\nWind speed: 6.4 m/s, direction: 305°\nHumidity: 6%\nTemperature: \n  - Current: 38.78°C\n  - High: 38.78°C\n  - Low: 38.78°C\n  - Feels like: 35.41°C\nRain: {}\nHeat index: None\nCloud cover: 0%'

Conclusion

If you made it so far Private. You Deserve it!!

All jokes apart this covers everything there is to the basics of LangGraph. Now go on and build some amazing AI Agents

In the upcoming blogs, we will be covering more advanced concepts of LangGraph

In this series, we will be making a discord bot that talks like your favorite character. We will first learn how to collect our own data, Fine-Tune a Transformer Model on this custom data, use this model as an inference API and finally deploy it as a discord Bot that you can play around with and show off to your friends!

Inspired by one of my favorite sci-fi shows Rick and Morty we will be making a Rick chatbot. But you can do this for any character and bring them to life

In this Part 2 we will be seeing the Basic theory of the Transformer Architecture and use the dataset that we made in Part 1 to fine-tune a Transformer Model and deploy this model onto Hugging Face🤗

But Why Transformer

The famous paper “Attention is all you need” in 2017 changed the way we were thinking about attention. With enough data, matrix multiplications, linear layers, and layer normalization we can perform state-of-the-art-machine-translation.

The Transformer Architecture is our first attempt at the unification of different fields of Deep Learning like Natural Language Processing , Computer Vision , Reinforcment Learning

In recent years, the transformer model has become one of the main highlights of advances in deep learning .This tweet by Andrej Karpathy summarizes the Transformer Architecture nicely

Over the years Transformer Models have evolved and you may have been using them with out even knowing about it . Some of the highlights include

• The original Transformer breaks previous performance records for machine translation.

• BERT popularizes the pre-training then finetuning process, as well as Transformer-based contextualized word embeddings. It then rapidly starts to power Google Search and Bing Search.

• GPT-2 demonstrates the machine’s ability to write as well as humans do.

• First T5, then T0 push the boundaries of transfer learning (training a model on one task, and then having it do well on other adjacent tasks) and posing a lot of different tasks as text-to-text tasks.

• GPT-3 showed that massive scaling of generative models can lead to shocking emergent applications (the industry continues to train larger models like Gopher, MT-NLG…etc).

So next time if some one asks you about GPT Models you know that these state of the art model are nothing but transformer decoder block stacked on each other!!!

If you want to dig deeper into the nitty-gritty of the transformer Architecture and the maths behind it I would recommend everyone to check these amazing Blogs

• AI Summer

• Jalammar.github.io

T5 Transformers

A T5 is an encoder-decoder model. It converts all NLP problems like language translation, summarization, text generation, question-answering, to a text-to-text task.

The original T5 paper, proposes reframing all NLP tasks into a unified text-to-text-format where the input and output are always text strings, in contrast to BERT-style models that can only output either a class label or a span of the input.

The text-to-text framework allows it to use the same model, loss function, and hyperparameters on any NLP task, including machine translation, document summarization, question answering, and classification tasks (e.g., sentiment analysis). We can even apply T5 to regression tasks by training it to predict the string representation of a number instead of the number itself.

Fine Tuning on Custom Dataset

For fine-tuning this model on our dataset, we will require high computing power. So if you have a GPU like RTX 2060 on your local machine you can use that. But even if you don't have a high-end machine you can still fine-tune the model using Google Colab for free , which we will be using in this Blog

What is Fine Tuning

Fine-tuning a pre-trained language model (LM) has become the de facto standard for doing transfer learning in natural language processing.

In the standard transfer learning setup , a model is first pre-trained on large amounts of unlabelled data using a language modelling loss such as masked language modelling (MLM; Devlin et al., 2019). The pre-trained model is then fine-tuned on labelled data of a downstream task using a standard cross-entropy loss.

For the purpose of this tutorial going through each step will take long time so we will be using a Python Library SimpleT5 that abstracts away a lot of this process.

Fine Tuning Using SimpleT5

simpleT5 is built on top of PyTorch-lightning⚡️ and Transformers🤗 that lets you quickly train/fine-tune T5 models.

With simpleT5 — It is very easy to fine-tune any T5 model on your dataset (Pandas dataframe )— for any task (summarization, translation, question-answering, or other sequence-to-sequence tasks), just — import, instantiate, download a pre-trained model and train.You can learn more about SimpleT5 in the docs

Let's fine tune our chatbot :

Processing Data

We can use the dataset we made in Part 1 or simply download it from any Platforms like Kaggle or HuggingFace as a CSV File

But simpleT5 expects a pandas dataframe as a dataset with 2 columns — source_text and target_text .

The dataset I am using is the RickAndMortyScripts.csv from Part 1 and we can simply convert it into a pandas dataframe

For the context of making a chatbot that talks like Rick our target_text can be the lines by rick .

And source_text can be all the lines said just earlier to the given line . eg . If target_text is the line at index = 2 then source text will be the line at index = 1

context_x = []
response_y = []

#excluding i=0 bcz i-1 will be out of bound
for i in range(1 , len(df)):
  if df['name'][i] == 'Rick' :
    context_x.append(df['line'][i-1])
    response_y.append(df['line'][i])

We can convert these lists into pandas dataframe

col = [ 'source_text' , 'target_text']

data = pd.DataFrame(list(zip(context_x, response_y)),
               columns =['source_text', 'target_text'])

Next Let's split our dataset into train and test - and we're done

from sklearn.model_selection import train_test_split

train_df , test_df = train_test_split(data , test_size = .2)

Now our training dataset looks some thing like this

Training

First of all we need to install SimpleT5 package

pip install simplet5

We will import SimpleT5 class and download a pre-trained T5 model

from simplet5 import SimpleT5

model = SimpleT5()
model.from_pretrained("t5","t5-base")

Finally Load the dataset in the model and train it

model.train(train_df=train_df, # pandas dataframe with 2 columns: source_text & target_text
            eval_df=test_df, # pandas dataframe with 2 columns: source_text & target_text
            source_max_token_len = 256, 
            target_max_token_len = 256,
            batch_size = 8,
            max_epochs = 12,
            use_gpu = True,
            outputdir = "outputs",

            )

• source_max_token_len and target_max_token_length control the length of the encoded token of source and target text

You can tweak around with hyper-parameters to see which performs better for your use case

Now grab a cup of Coffee because training is gonna take some while . Training with a Colab Free tier on a Tesla T4 took me around 20 minutes to train.

Inference

Now that the model is trained now it's to yield it's power

In the output directory we will have folders containing the model's weight at each epoch . load the one with the minimum loss

model.load_model("t5" , "/content/outputs/simplet5-epoch-11-train-loss-1.7831-val-loss-3.6891" , use_gpu=True)

model.predict("are we in living in a simulation?")

Now you can go crazy with what you want to ask to your favourite character🎉

Deploying the Model

We will be deploying this model onto HuggingFace🤗 where every one can play around with you model

Create a new repository on your Hugging Face Account

Connect your Colab Environment to your Hugging Face Account

!pip install huggingface_hub

Generate a Access Token for your Hugging Face Account and keep it with yourself to login

from huggingface_hub import notebook_login

notebook_login()

Push your files onto your repository

from huggingface_hub import HfApi
api = HfApi()

api.upload_file(
    path_or_fileobj="/content/outputs/simplet5-epoch-11-train-loss-1.7831-val-loss-3.6891/tokenizer_config.json",
    path_in_repo="tokenizer_config.json",
    repo_id="Prarabdha/T5-Transformer-RickBot",
    repo_type="model",
)

• path_or_fileobj : Path to the file that you are pushing

• path_in_repo : Path of the file in your repository

• repo_id : is just Your_Username/Name_of_repository

• repo_type : model or dataset

Push all your files in the folder with least loss onto Hugging Face one by one

Now your repo should look something like this

** Without the T5-Transformer-RickBot folder in the repo (that was a mistake)

Voila — You’re done !🎉

Now you have successfully deployed you model on Huggin Face Hub it can be used by developers across the globe

Without you spending hours and hours on trying to make inference API on you own and trying to host it on something like AWS

Have fun chatting with your favourite character!!🤩

Check Out RickBot ❤️

I hope that you found this tutorial helpful. See you in the next part of this series and Happy Coding!!

In this series, we will be making a discord bot that talks like your favorite character. We will first learn how to collect our own data, Fine-Tune a Transformer Model on this custom data , use this model as an inference API and finally deploying it as a discord Bot that you can play around with and show off to your friends!

Inspired by one of my favorite sci-fi shows Rick and Morty I we will be making a Rick chatbot . But you can do this for any character and bring them to life

In this Part 1 we will be making our own dataset by scrapping the internet with the help of BueatifulSoup , conveting the dataset into a CSV File and finally publishing it

But Why Data?🤔

Data is the most important and must-have food for machine learning. It can be any fact, text, symbols, images, videos, etc., but in unprocessed form. Machine learning without data is nothing but a bare machine with no soul and no mind. This data makes machines do such amazing tasks, which we have not thought of a few years back in history.

Most of us don't pay any attention to the kind of data we are using even though our Quality of data dictates the Quality of our model

If I would ask you how many projects did you abandon , just because you couldn't find a dataset for your use case ?

Atleast for me there would be so many of them that I couldn't complete just because there was no dataset on any platforms like Kaggle and HuggingFace

Web Scraping using Beautifulsoup🤗

In this section, we will be learning how to create a web scraper in Python. You will learn how to inspect a website to prepare for scraping, extract specific data using BeautifulSoup

For our project, we can get the transcripts of almost all episodes of Rick and Morty on these 2 sites -

• RickandMorty.fandom.com

• RickandMorty.newtfire.org

When scraping data we will need to get a basic idea of HTML code

import requests
topic_url = "https://rickandmorty.fandom.com/wiki/Rickipedia"
response = requests.get(topic_url)

Request Library helps us to extract the HTML code of any web page and response will return the complete code as a string

But getting a large string of code is not very helpful to us so we can use a python package BeautifulSoup. To install it, just run this command on your terminal:

pip install beautifulsoup4

Now we can use BeautifulSoup to wrap our HTML Code and send in our queries

from bs4 import BeautifulSoup
doc = BeautifulSoup(page_contents , 'html.parser')
topics_link_tags = doc.find_all('a' , {'class' : 'image link-internal'})
len(topics_link_tags)

This code helps us to find all the anchor tags in our code that have a class name as 'image link-internal' . We can send in all types of queries to search tags by ids or extract their inner text to read further you can see the docs . But we will be covering all the commonly used functions in this blog

Journey Begins 😎

If you go to the RickandMorty.fandom.com site and scroll you can see that the transcripts we need are not here. We need to open the 'Episodes' link in this table

To extract this link we need an understanding of HTML written on this page. You can simply use the inspect tool on chrome to get the code

We will need to extract the link present inside the anchor tag wrapping the highlighted image tag , in order to scrape the next page

#Since the anchor we want to extract is 9th on the page
episode_url = 'https://rickandmorty.fandom.com' + topics_link_tags[8]['href']
response_episode = requests.get(episode_url)
episode_page_content = response_episode.text
doc_episode = BeautifulSoup(episode_page_content , 'html.parser')

On the next page we have the link to all the episode and on further inspection of these individual links we can get transcripts

Plan of Action

• Get a strategy to extract transcript on the individual episode

• Store the Data

• Repeat it on every episode page

Extracting transcripts for a Individual Page

On the transcript page of the one of the episode we can see that each line is wrapped inside <paragraph>tags . The name of Character is either in<bold> tag or <anchor> tag and the line by that character is written outside these tags

Plan of Action

• loop over all paragraph> tags present on that page

• Find the <b> tag inside a <p> tag and check for <a> tag in it

• If no <a> tag present inside <b> , then content of <b> is name of character , otherwise go for<a>

• Extract the line by that character

We can define a simple function that can parse the HTML for every page extract the data and store it in array

def page_transcript(link , episode_num) :

    data = []

    response_episode_page = requests.get(link)
    page_content = response_episode_page.text

    doc_episode_page = BeautifulSoup(page_content , 'html.parser')
    para_tags = doc_episode_page.find_all('p')

    for tag in para_tags :
        try :
            if tag.find('b') == None:
                continue
            else :
                list_child = list(tag.children)
                dialouge = list_child[1]

                if list_child[0].find('a') == None :
                    name = list_child[0].contents[0]
                    # print(list_child[0].contents[0])

                else :
                    name = list_child[0].find('a').contents[0]
                    # print(list_child[0].find('a').contents[0])

        except Exception as e:
            continue

        data.append([episode_num , name , dialouge])

    return data

There may be a better way to write the above code you are free to explore and play around with the code

Extracting Data from every transcript page link

To navigate to the transcript page we need to go through each link of episode contained inside the season table and further add '/Transcript' in that episode URL

In this code we loop over all the tables, extract the links of episodes and call our page_transcript() function to store the data for every episode

dataset = []

j = 0
for i in range(1 , 6):
    table = doc_episode_tables[i]
    anchor_tags = doc_episode_tables[i].find_all('a')

    for tags in anchor_tags :

        try :
            dataset_episode_title.append(tags['title'])
            link = 'https://rickandmorty.fandom.com' +tags['href'] + '/Transcript'
            j+=1
            print(link , j)
            data = page_transcript(link=link , episode_num=j)
            dataset.append(data)

        except Exception as e:
            continue

Finally! You have the dataset list containing the transcripts for all the episodes🎉

But our work is not done yet!!!

Common Errors

In general when scrapping data this way we make a bet that code will be standardized for every page in the website but that is always not the case

We come across this problem with this website many of the transcript pages have totally different ways of writing transcripts in HTML code or sometimes even data is completely missing. Which will result in Low-Quality Data.So we will have to manually check for the faulty pages and come up with scripts to solve them

1)All content put in one p tag error

In this rather than putting the name of the character in a <b> or <a> tag it is simply put in a <p> tag with the lines of the character

In this code, we are extracting the transcript of episode 24

url_error = "https://rickandmorty.fandom.com/wiki/Pickle_Rick/Transcript"

response_error = requests.get(url_error)
content_error = response_error.text
doc_episode_error = BeautifulSoup(content_error , 'html.parser')

para_tag_error  = doc_episode_error.find('p')

text_error = para_tag_error.get_text()
text_error = text_error.rsplit("\n")

for text_e in text_error :
    try:
        text_e_i = text_e.rsplit(': ')
        dataset[23].append([24 , text_e_i[0] , text_e_i[1]])
    except Exception as e:
        continue

2)Retriving missing content from other site

We solve this problem by getting the missing content from a different site RickandMorty.newtfire.org

In this code we extract the transcript for episode 27 and by simply changing the ID of div in the page we can do the same for different episodes

url_error_2 = "http://rickandmorty.newtfire.org/transcripts.html"
response_error_2 = requests.get(url_error_2)
content_error_2 = response_error_2.text
doc_episode_error_2 = BeautifulSoup(content_error_2 , 'html.parser')

opt_26 = doc_episode_error_2.find('div' , id = 'aboutopt26')

opt_26_speaker = opt_26.find_all('span' , {'class':'speaker'})
opt_26_speech = opt_26.find_all('span' , {'class':'speech'})
for i in range (0 , len(opt_26_speaker)):
     dataset[26].append([27 , opt_26_speaker[i].get_text() , opt_0_speech[i].get_text()])

Exporting Dataset as CSV File 🙌

Finally after correcting a lot of errors our dataset is finally readyand we can export it into a CSV File

In this code we will first convert this python list into pandas dataframe and finally convert it to a csv file

import pandas as pd

col = ['episode no.' , 'speaker' , 'dialouge']
df = pd.DataFrame(dataset[0] , columns=col)

for i in range(1 , 51) :
    df_temp = pd.DataFrame(dataset[i] , columns=col)
    df = df.append(df_temp , ignore_index = True)

df.to_csv('Rick&Morty.csv')

Share your work in Public 🤩

Now that you have created your dataset, it wouldn't be fun just to keep it with yourself . You can Publish it on platforms like HuggingFace and Kaggle so that it can be used by anyone

You can simply do that by creating a new dataset on any of these of platforms and importing your CSV File and that's it. Make Sure to add good Documentation for your dataset to make it easy to use for others.

That's it you just made your Dataset publicly available which can now be used by everyone

You can find the Rick And Morty dataset on Kaggle or HuggingFace make sure to use it in your next project and give an upvote!

I hope that you found this tutorial helpful. See you in the next part of this series and Happy Coding!!

In this article we are gonna cover the following topics -

1. Maths Behind SVD
2. TensorGame
3. Challenges posed by TensorGame
   • Synthetic demonstrations
   • Change of basis
   • Data augmentation
   • Neural network architecture
4. Conclusion

Singular Value Decomposition (SVD)

SVD is one of the most important algorithms in linear algebra. You can think of it as a tool that helps in data reduction of Big Data eg. high-resolution images or videos. In the last generation of computational science, we used may be things like the Fourier transform, Bessel's function, and spherical harmonics to map the system of interest to new and simpler co-ordinate. But for example, if we have a complex system like turbulent flow over a Boeing wing, there is no off-the-shelf transformation to cater to this problem but a technique like SVD can be tailored to any general problem This algorithm is literally used all over the place like compression of large data, Google's page rank algorithm, Facebooks's face recognition algorithms and recommendation systems at Netflix, and Amazon to find correlation patterns

Now that I have got you hyped enough let's dive in

Let's assume we have image with a million pixels and it is vectorized to a column vector m x 1 and we use n such images putting them togather to get a m x n matrix where m>>>n (m is in order of millions and n is no.of image in order of thousands) Then that matrix can be represented as

where U , V are orthogonal matrices which is represented in the above image U is of same dimension as of M but with some distortionand that's why you can guess it is called the eigenvector and the sigma matrix is a diagonal matrix till n x n matrix and other than that it's all zero where all enteries are in hierichal order describing the importance of first row of U and first column of V in representing our data M

But we can approximate our data M by just using first k columns in our matrices U , V , W

To get a even deeper understanding of SVD I would highly recommend this youtube playlist by Steve Brunton

AlphaTensor

When we talk about efficient algorithms the efficiency doesn’t come from thin air it comes by optimizing our algorithm to the kind of hardware we have . The GPUs or TPUs we use take a lot time in multiplication than addition so if we could reduce the number of multiplication operation by some kind of manipulation and replace it with addition . You all must remember this from your high school mathematics

As you can see with a simple manipulation we reduce the number of multiplications from 2 to 1 thus optimizing our calculation and when we do some similar things in a 2 x 2 matrix multiplication we can optimize matrix multiplication

As you can clearly see there can be many combination that we can form with trial and error to get our required result and this is where reinforcement learning come where our agents learns the suitable parameters in form of a game Now we are gonna look at the structure of this game

Let’s assume we have two 2 x 2 matrices to multiply (A & B) to get a resultant matrix C

Tensor T2 representing the multiplication of two 2 × 2 matrices. Tensor entries equal to 1 are depicted in purple, and 0 entries are semi-transparent. The tensor specifies which entries from the input matrices to read, and where to write the result. For example, as c1 = a1b1 + a2b3, tensor entries located at (a1, b1, c1) and (a2, b3, c1) are set to 1

Then if we could do a matrix decomposition of T2 we can find the values of parameter let me unroll this Looking at the given state our agent makes a policy i.e choses a set of triplet (u(t) , v(t) , w(t) ) Let’s assume agent chooses the given combination of u , v , w taking a outer product would give us

Taking outer product of these vectors would give us ,

where above picture represents a 4x4x4 tensor with the same 4x4 matrix repeated 4 times in z direction is the rank 1 approximation of the data M

Now if we keep coming up with good values to the triplet u , v, w make 3d tensors with them then add them we shall get back our initial T2

TensorGame

The state of TensorGame after step t is described by a tensor St and initially S0 = T2 then agent finds a worthy triplet find out it’s outer product then subtract the result with the current state , again use that new state and new triplets to find next state and so on . And the game is finally won if after 7 approximation of u , v , w triplet we get the final state as zero which is what it should be if you remember the maths from previous section

• For every step taken, we provide a reward of −1 to encourage finding the shortest path to the zero tensor.
• , we constraint {u(t), v(t), w(t)} to have entries in a user-specified discrete set of coefficients F (for example, F = {−2, −1, 0, 1, 2}).
• TensorGame is played by the agent AlphaZero1, which achieved superhuman performance in the classical board games of Go, chess and shogi, Similarly to AlphaZero, AlphaTensor uses a deep neural network to guide a Monte Carlo tree search (MCTS) planning procedure. The network takes as input a state (that is, a tensor St to decompose), and outputs a policy and a value.

challenges posed by TensorGame

The main challenge posted by TensorGame is that of a enormous action space . We can overall improve the performance over aplain AlphaZero agent by following methods

1. Synthetic demonstrations

Although decomposition of matrices is a difficult task but the task of making matrices from randomly choosen {u(t), v(t), w(t)} and adding them up to get a matrix is quite easy

Then we can train the network on this diverse set of examples

2. Change of basis

Tn (Fig. 1a) is the tensor representing the matrix multiplication bilinear operation in the canonical basis. The same bilinear operation can be expressed in other bases, resulting in other tensors. To know more about basis you can see this amazing video by 3Blue1Brown

These different tensors are equivalent they have the same rank, and decompositionsobtained in a custom basis We leverage this observation by sampling a random change of basis at the beginning of every game, applying it to Tn, and letting AlphaTensor play the game in that basis . This crucial step injects diversity into the games played by the agent.

3. Data augmentation

From every played game, we can extract additional tensor-factorizationpairs for training the network.Specifically, as factorizations are order invariant (owing to summation), we build an additional tensor-factorization training pair by swapping a random action with the last action from each finished game.

4. Neural network architecture

The network broadly consists of the following components Input, a torso, followed by a policy head that predicts a distribution over actions, and a value head that predicts a distribution of the returns from the current state

4.1 ) Input - The model is given all relevant information about the current state and the previous state . We plug in the the current state St as a tensor and last h actions (h being a hyperparameter usually set to 7) as a scalar

4.2 )Torso - It basically maps the tensors and scalars from the input to a representation that is useful to both policy and value heads It is a architecture based on modification of transformers and its main signature is that it operates over three S × S grids projected from the S × S × S input tensors.

4.3 )Policy Head - Using the embedding generated by the torso network , this architecture gives a possible move for the agent i.e. choosing a correct set of {u(t), v(t), w(t)} given the previous state . The network makes the use of various attention mechanisms form transformers architecture you can learn more about it from this amazing blog

4.4 )Value Head - The value network assigns value/score to the state of the game by calculating an expected cumulative score for the current state s . Actions that result in good state gets higher score . The key objective of our agent is to maximize this score

Conclusion

AlphaTensor’s algorithm improves on Strassen’s two-level algorithm in a finite field for the first time since its discovery 50 years ago.

AlphaTensor also discovers a diverse set of algorithms with state-of-the-art complexity – up to thousands of matrix multiplication algorithms for each size, showing that the space of matrix multiplication algorithms is richer than previously thought . These algorithms multiply large matrices 10-20% faster than the commonly used algorithms.

Because matrix multiplication is a core component in many computational tasks, spanning computer graphics, digital communications, neural network training, and scientific computing, AlphaTensor-discovered algorithms could make computations in these fields significantly more efficient.

Prerequisites

To make anything on blockchain you should have a decent grab over web2 topics

• Very basic knowledge of HTML & CSS
• Knowledge of Asynchronous programming in Javascript
• Basic idea of smart contracts and solidity

For this tutorial we will be using metamask as our digital wallet and remix IDE for compiling and deploying our smart contract

Now the FUN Starts

1. Setting up Metamask

• Download the Metamask wallet extension if you haven't
• Switch to Ropsten Testnet from the Ethereum Mainnet by clicking on the top

If you are working with metamask for first time then probably you will have 0 etheres on any testnet but here is the amazing thing since these are testnet ethers on this network have no valueso you can get it for free from any faucet by entering your metamask wallet address Faucet link to request funds

2. Create and Serve a Simple Webpage

The first step is to create a basic HTML page.

• Create a new folder (directory) in your terminal using mkdir
• In a code editor (e.g. Atom, or Visual Studio Code), open the folder
• Create a new file called index.html
• Open index.html
• Create HTML boilerplate
• We will need a label, an input, and buttons.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>My first Dapp</title>
</head>
<body>
    <div class="">
        <h1>This is my first Dapp</h1>
        <p>Here we can either set the mood or read the mood</p>
        <button onclick="getMood()">Get Mood</button> <br>
        <input type="text" id="mood" placeholder="Input Mood">
        <button onclick="setMood()">Set Mood</button>
    </div>

</body>
</html
>

OPTIONAL: Create a new file index.css , add some styles to make it look nicer

body {
    text-align: center;
    font-family: Arial, Helvetica, sans-serif;
  }

div {
    width: 20%;
    margin: 0 auto;
    display: flex;
    flex-direction: column;
}

button {
    width: 100%;
    margin: 10px 10px 5px 0px;
    border-radius: 2rem;
}

Feel free to add more styling as to be honest ui looks kinda ugly

• Install a http server. Use any you like, but we recommend lite-server for beginners:

npm install -g lite-server #install lite-server globally

• Serve the webpage via terminal/command prompt from the directory that has index.html in it and run:

lite-server

• Go to http://127.0.0.1:3000/ in your browser to see your page!

Your front end is now complete!

3. Create a Basic Smart Contract

For this tutorial we will be using Remix IDE

Make a new file mood.sol in the file structure and specify the solidity version you want to use

pragma solidity >=0.7.0 <0.9.0;

Create a contract moodDiary

pragma solidity >=0.7.0 <0.9.0;

contract moodDiary{

}

Define a string variable that stores the value of mood and a function getMood() that returns it's value

pragma solidity >=0.7.0 <0.9.0;

contract moodDiary{

      string mood;

      function getMood() public view returns(string memory){
        return mood;
    }
}

Also define a function that can change the state of variable mood on the blockchain i.e. setMood()

pragma solidity >=0.7.0 <0.9.0;

contract moodDiary{

      string mood;

      function getMood() public view returns(string memory){
        return mood;
    }

      function setMood(string memory _mood) public{
        mood = _mood;
    }
}

Compile the solidity file

Deploying on Ropsten Network

Make sure that you are on Ropsten testchain on your Metamask wallet on the Deploy/Run section of remix select Injected Web3 as encironment

Click on Deploy button It will take some time so take a sip of your coffee as this transaction is being added to the blockchain

Remember to copy:

• The deployed contract's address Copy it via the copy button next to the deployed contracts pulldown in remix's Run tab

• The contract ABI Copy it via the copy button under to the contract in remix's Compile tab (also in Details)

4.Connect your Webpage to your Smart Contract

Back in your local text editor in index.html, add the following code to your html page:

Import the Ethers.js source into your index.html page inside a new set of script tags:

<script src="https://cdn.ethers.io/lib/ethers-5.2.umd.min.js"
type="application/javascript"
></script>

<script>
  ////////////////////
  //ADD YOUR CODE HERE
  ////////////////////
</script>

Inside the script tag, import the contract ABI (what is that?) and specify the contract address on our provider's blockchain:

```const MoodContractAddress = ""; const MoodContractABI = let MoodContract; let signer;

For the contract ABI, we want to specifically navigate to the JSON Section. We need to describe our smart contract in JSON format

```const MoodContractABI = [
    {
        "constant": true,
        "inputs": [],
        "name": "getMood",
        "outputs": [
            {
                "internalType": "string",
                "name": "",
                "type": "string"
            }
        ],
        "payable": false,
        "stateMutability": "view",
        "type": "function"
    },
    {
        "constant": false,
        "inputs": [
            {
                "internalType": "string",
                "name": "_mood",
                "type": "string"
            }
        ],
        "name": "setMood",
        "outputs": [],
        "payable": false,
        "stateMutability": "nonpayable",
        "type": "function"
    }
]

Next, Define an ethers provider. In our case it is Ropsten:

const provider = new ethers.providers.Web3Provider(window.ethereum, "ropsten");

Request access to the user's wallet and connect the signer to your metamask account (we use [0] as the default), and define the contract object using your contract address, ABI, and signer

provider.send("eth_requestAccounts", []).then(() => {
  provider.listAccounts().then((accounts) => {
    signer = provider.getSigner(accounts[0]);
    MoodContract = new ethers.Contract(
      MoodContractAddress,
      MoodContractABI,
      signer
    );
  });
});

Create asynchronous functions to call your smart contract functions

async function getMood() {
  const getMoodPromise = MoodContract.getMood();
  const Mood = await getMoodPromise;
  console.log(Mood);
}

async function setMood() {
  const mood = document.getElementById("mood").value;
  const setMoodPromise = MoodContract.setMood(mood);
  await setMoodPromise;
}

Test Your Work Out!

• Got your webserver up? Go to http://127.0.0.1:3000 or http://localhost:3000/ in your browser to see your page!
• Test your functions and approve the transactions as needed through Metamask. Note block times are ~15 seconds... so wait a bit to read the state of the blockchain
• See your contract and transaction info via https://ropsten.etherscan.io/
• Open a console (Ctrl + Shift + i) in the browser to see the magic happen as you press those buttons

DONE!

Celebrate! You just made a webpage that interacted with a real live Ethereum testnet on the internet! That is not something many folks can say they have done!