Durgesh Tiwari
Author
Building an AI application is more than simply connecting a chatbot to an AI model.
A real AI application can include several components working together, such as an application backend, LLM API, AI gateway, model routing, embeddings, vector databases, search systems, RAG, and model serving.
For example, an AI customer support chatbot may receive a user's question, find relevant information, send that context to an AI model, and return the generated answer.
Understanding the role of each component helps in designing and building reliable AI applications.
AI application architecture describes how the different components of an AI application are arranged and how they communicate with each other.
A simple AI application may include:
User interface.
Application backend.
LLM API.
AI model.
Database.
Monitoring.
More advanced applications may also use:
AI gateways.
Model routing.
Embeddings.
Vector databases.
Semantic search.
RAG.
Model serving.
Streaming.
Asynchronous processing.
In simple words, AI application architecture shows how a user's request moves through the application and how the AI generates a response.
User
↓
Web / Mobile Application
↓
Application Backend
↓
LLM API / AI Model
↓
AI Response
↓
UserNot every AI application needs all these components.
A simple chatbot may only need an application backend and an LLM API. A larger application may also use RAG, vector search, multiple models, monitoring, and background processing.
Suppose you are building an AI assistant for an online shopping website.
A customer asks:
"Can I return a product after 15 days?"The application may:
Receive the customer's question.
Search the company's return policy.
Find the relevant information.
Send the information to the AI model.
Generate a clear answer.
Show the answer to the customer.
This example shows why AI application architecture matters: each component has a specific role in processing the request and generating the final response.
An LLM API allows an application to communicate with a Large Language Model.
Instead of running the model directly on your own computer or server, an application can send requests to an AI provider through an API. The model processes the request and returns a response.
In simple words, an LLM API is a way for an application to use an AI language model through the internet.
The basic process is:
User
↓
Application
↓
LLM API Request
↓
Language Model
↓
Generated Response
↓
Application
↓
UserThe application sends the required information to the API, and the API returns the model's output to the application.
Example
Suppose a user enters:
"Explain Kubernetes in simple words."The application sends this prompt to an LLM API.
The language model processes the request, generates an answer, and sends the response back to the application. The application then displays the answer to the user.
Depending on the application, an API request may include:
User prompt.
System instructions.
Previous conversation.
Retrieved information.
Model name.
Generation settings.
For example:
System Instructions
+
User Question
+
Retrieved Information
↓
LLM API
↓
AI ResponseLLM APIs make it easier to add AI capabilities to applications without managing the underlying model infrastructure.
Common use cases include:
AI chatbots.
Content generation.
Question answering.
Text summarization.
Translation.
Code generation.
Customer support.
Document analysis.
An AI Gateway is a layer between an application and one or more AI models or providers.
Instead of connecting an application directly to each model, the gateway provides a common interface for managing AI requests.
Application
↓
AI Gateway
↙ ↓ ↘
Model A Model B Model CIn simple words, an AI Gateway provides one place to manage requests to multiple AI models.
An application may use different models for different tasks. For example, one model may handle simple questions, another may handle complex tasks, and another may generate embeddings.
Without a gateway, the application may need to manage separate connections and configurations for each model or provider.
An AI Gateway can help manage:
Authentication.
Model access.
Request routing.
Rate limits.
Usage tracking.
Logging.
Monitoring.
Error handling.
Example
Suppose an application uses:
Model A → Simple questions
Model B → Complex tasks
Model C → EmbeddingsThe application can send requests through the gateway:
Application
↓
AI Gateway
↓
Correct AI ModelThe gateway can route each request to the appropriate model based on the configured rules.
This can make applications easier to manage when they use multiple AI models or providers.
Model routing is the process of choosing the most suitable AI model for a particular request.
Different models can have different strengths. Some may be:
Faster.
Cheaper.
Better at reasoning.
Better at coding.
Better at handling long inputs.
Better suited for specific tasks.
Because of this, using the same model for every request may not always be the most efficient approach.
In simple words, model routing decides which AI model should handle a particular request.
Example
Suppose an application receives two requests:
Question 1:
"Translate this sentence into Hindi."
Question 2:
"Explain this complex software architecture."The application could route them to different models:
Simple Request
↓
Fast / Lower-Cost Model
Complex Request
↓
More Capable ModelA basic model-routing process looks like this:
User Request
↓
Model Router
↓
Analyze Request
↓
Choose Model
↓
Selected AI Model
↓
ResponseThe router can use rules, request type, model capabilities, cost, latency, or other configured criteria to select a model.
Model routing can help applications:
Reduce AI costs.
Improve response speed.
Use powerful models only when needed.
Support multiple AI models or providers.
Match models to different tasks.
For larger AI applications, model routing can improve the balance between cost, speed, and model quality.
An embedding is a numerical representation of information, such as text.
An embedding model converts text into a list of numbers called a vector.
For example:
"How do I reset my password?"
↓
Embedding Model
↓
[0.21, -0.43, 0.78, ...]These numbers are not usually meaningful to humans. Applications use them to compare how similar different pieces of information are.
In simple words, embeddings turn text into numbers so an application can compare the meaning of different texts.
Example
Consider these two questions:
"How can I reset my password?"
"I forgot my password. How can I change it?"The wording is different, but both questions have a similar meaning.
Their embeddings can therefore be close to each other in vector space. This allows an application to find relevant information based on meaning rather than only matching exact words.
Semantic similarity means comparing texts based on their meaning.
For example:
"How do I change my password?"
↕
"How can I reset my password?"These sentences use different words but have a similar meaning, so their embeddings may be close together.
Embeddings are commonly used for:
Semantic search.
Similarity search.
Recommendation systems.
Document search.
RAG applications.
Text classification.
Embeddings are especially useful when an AI application needs to search and compare a large amount of information based on meaning rather than exact keywords.
A vector database is a database designed to store and search vectors, such as embeddings.
Traditional databases are commonly used for structured data such as:
Name
Email
Age
Product ID
PriceVector databases are useful when an application needs to find information based on similarity or meaning rather than exact keyword matches.
First, documents are converted into embeddings and stored as vectors.
Documents
↓
Embedding Model
↓
Vectors
↓
Vector DatabaseWhen a user asks a question, the question is also converted into an embedding.
User Question
↓
Query Embedding
↓
Vector Database
↓
Similar InformationThe database compares the query vector with stored vectors and returns the most similar results.
Example
Imagine a company has 10,000 customer support documents.
A customer asks:
"How can I get my money back?"The application converts the question into an embedding and searches the vector database.
It may find a document such as:
"Customer Refund Policy"even if that document does not contain the exact phrase:
"How can I get my money back?"This is because the search is based on semantic similarity, not just exact keyword matching.
Vector databases are commonly used for:
Semantic search.
Similarity search.
RAG applications.
Document search.
Recommendation systems.
AI-powered question answering.
In simple words, a vector database stores embeddings and helps AI applications find information that is similar in meaning to a user's query.
Semantic search finds information based on the meaning of a query, rather than relying only on exact keyword matches.
This is useful because people can ask the same question using different words.
Example
A user searches:
"How do I change my password?"A document may contain:
"Steps for resetting your account credentials"A traditional keyword search may not find this document because the wording is different.
Semantic search can identify that both are related to changing or resetting a password.
A common semantic search process is:
User Query
↓
Embedding Model
↓
Query Vector
↓
Vector Database
↓
Similar Documents
↓
Search ResultsThe query is converted into an embedding, and the resulting vector is compared with stored document vectors to find relevant information.
Keyword Search | Semantic Search |
|---|---|
Mainly looks for matching words | Looks for similar meaning |
Works well for exact terms | Works well with natural-language questions |
May miss differently worded information | Can find differently worded information |
Usually simpler to implement | Commonly uses embeddings and vector search |
RAG stands for Retrieval-Augmented Generation. It is a technique that allows an AI application to retrieve relevant information from an external knowledge source and provide that information to an LLM before generating a response.
External sources can include:
Company documents.
Databases.
Knowledge bases.
Product documentation.
Internal policies.
In simple words, RAG helps an AI model answer questions using relevant information from external sources such as your own documents or database.
An LLM may not have access to an organization's private or up-to-date information.
For example, suppose a company has an internal document:
Employee Leave PolicyA general AI model may not know the company's latest leave policy.
With RAG, the application can retrieve the relevant information from the company's knowledge base and provide it to the LLM.
User Question
↓
Retrieve Relevant Information
↓
Knowledge Base
↓
Relevant Context
↓
LLM
↓
Generated AnswerA typical RAG process includes two main steps:
1. Retrieval — The application searches a knowledge source and finds information relevant to the user's question.
2. Generation — The retrieved information is provided to the LLM as context, and the model generates an answer based on that context.
User Question
↓
Search Knowledge Base
↓
Retrieve Relevant Information
↓
Send Context + Question to LLM
↓
Generate AnswerExample
An employee asks:
"How many days of parental leave can I take?"The RAG system searches the company's HR documents and retrieves the relevant leave policy.
The retrieved information is then provided to the LLM along with the employee's question.
The LLM uses that context to generate the answer.
RAG does not require the LLM to be retrained every time a document changes.
Instead, the application retrieves the latest relevant information when the user asks a question and provides it to the model as context.
This makes RAG useful for applications that work with frequently changing or private information.
RAG is commonly used for:
Company knowledge bases.
Customer support.
Technical documentation.
Internal AI assistants.
Research tools.
Legal document search.
Product documentation.
Enterprise chatbots.
In simple words, RAG connects an AI model with external knowledge so it can generate answers using relevant information that may not be part of its original training data.
A typical RAG architecture has two main stages:
Document preparation — prepares information so it can be searched.
Question answering — retrieves relevant information and uses it to generate an answer.
Before users ask questions, documents are processed and stored in a searchable format.
Documents
↓
Chunking
↓
Embeddings
↓
Vector DatabaseChunking breaks large documents into smaller pieces called chunks.
Embeddings convert these chunks into numerical vectors that represent their meaning.
The vectors are then stored in a vector database for later retrieval.
When a user asks a question, the RAG system searches for relevant information.
User Question
↓
Query Embedding
↓
Retrieval
↓
Relevant Chunks
↓
Reranking
↓
Relevant Context
↓
LLM
↓
Final AnswerThe question is converted into an embedding, which is used to find relevant chunks from the vector database.
Reranking can then reorder the retrieved results so that the most relevant information is placed first.
The selected information is provided to the LLM as context, and the LLM generates the final answer.
The complete process can be understood as two connected flows:
Document Preparation
Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
↑
│
│
Question Answering
User Question
↓
Query Embedding
↓
Retrieval
↓
Reranking
↓
Relevant Context
↓
LLM
↓
AnswerThis architecture allows the application to retrieve relevant information from a knowledge base instead of relying only on the model's existing knowledge.
Example
Imagine you are building an AI assistant for a university.
The knowledge base contains:
Admission rules.
Course information.
Exam schedules.
Fee details.
Hostel rules.
A student asks:
"What is the last date for admission?"The RAG system converts the question into an embedding and searches the vector database for relevant information.
It may retrieve the university's admission document, which is then provided to the LLM as context.
The LLM uses that information to generate the answer for the student.
In simple words, RAG architecture prepares documents for search, retrieves the most relevant information for a question, and gives that information to an LLM to generate the answer.
Chunking is the process of breaking a large document into smaller sections called chunks.
An AI application usually does not need to search through an entire 500-page document for every question. Instead, the document is divided into smaller pieces so the system can retrieve only the information that is relevant to the user's question.
Large Document
↓
┌────┼────┬────┐
↓ ↓ ↓ ↓
C1 C2 C3 C4Each chunk can then be converted into an embedding and stored in a vector database for retrieval.
Example
Suppose an employee handbook contains:
Chapter 1 → Company Rules
Chapter 2 → Working Hours
Chapter 3 → Leave Policy
Chapter 4 → Benefits
Chapter 5 → SalaryThe application can divide the handbook into smaller chunks.
If a user asks:
"How many days of leave can I take?"The system can retrieve chunks containing information from the Leave Policy instead of searching the entire handbook.
Sometimes neighboring chunks can share a small amount of text.
For example:
Chunk 1 → ...information about leave eligibility...
Chunk 2 → ...leave eligibility and the approval process...This overlap can help preserve information that might otherwise be split between two chunks.
Good chunking can improve:
Search results.
Retrieval accuracy.
The relevance of context sent to the LLM.
Response quality.
Retrieval efficiency.
If chunks are too large, they may contain too much unrelated information.
If chunks are too small, important information may be separated from the context needed to understand it.
The best chunking approach depends on the type and structure of the documents being used.
In simple words, chunking breaks large documents into smaller, meaningful pieces so a RAG system can find and use the right information more efficiently.
Retrieval is the process of finding the most relevant information for a user's question.
In a RAG application, retrieval usually works by converting the user's question into an embedding and searching the stored document embeddings for similar content.
User Question
↓
Query Embedding
↓
Similarity Search
↓
Vector Database
↓
Relevant ChunksThe system can retrieve one or more chunks that are most relevant to the question.
Example
The user asks:
"What is the company's refund period?"The retrieval system may find:
Chunk 12 → Refund Policy
Chunk 18 → Cancellation Policy
Chunk 31 → Payment PolicyThe system then selects the most relevant chunks and passes them to the next stage of the RAG pipeline.
Retrieval often uses similarity search to compare the user's query embedding with document embeddings.
The closer the vectors are in the embedding space, the more likely the corresponding content is to be relevant to the query.
Retrieval is an important part of RAG because the LLM needs relevant information to generate a useful answer.
If the retrieval system returns unrelated or incomplete information, the LLM may receive poor context and produce a less accurate answer.
Good retrieval helps the application:
Find relevant information.
Reduce unrelated context.
Provide better context to the LLM.
Improve the quality of generated answers.
In simple words, retrieval finds the most relevant pieces of information from a knowledge base and provides them to the LLM for generating an answer.
Reranking is the process of reviewing retrieved search results and putting the most relevant results at the top.
In a RAG system, the initial retrieval step may return several possible chunks. A reranker then evaluates these results more carefully and reorders them based on their relevance to the user's question.
User Question
↓
Initial Retrieval
↓
20 Possible Chunks
↓
Reranker
↓
5 Most Relevant Chunks
↓
LLMThe goal is to provide the LLM with the most useful information instead of sending all retrieved results.
Example
A user asks:
"What is the refund period for online orders?"The initial retrieval may return:
1. General payment information
2. Refund policy
3. Shipping policy
4. Product cancellation rules
5. Online order refund rulesA reranker examines how closely each result matches the question.
It may reorder them like this:
1. Online order refund rules
2. Refund policy
3. Product cancellation rules
4. General payment information
5. Shipping policyThe application can then pass the highest-ranked results to the LLM as context.
Retrieval | Reranking |
|---|---|
Finds potentially relevant results | Reorders the retrieved results |
Usually works on a larger set of results | Focuses on the retrieved candidates |
Quickly identifies possible matches | Evaluates relevance more carefully |
Produces candidate chunks | Produces a better-ranked set of chunks |
Reranking can help:
Improve retrieval quality.
Put the most relevant information first.
Reduce less useful context.
Give the LLM better information.
Improve the quality of RAG responses.
In simple words, retrieval finds possible answers, while reranking decides which of those results are most relevant.
Fine-tuning is the process of training an already trained AI model on a specific dataset to improve its performance for a particular task, style, or behavior.
In simple words, fine-tuning teaches an existing AI model how to perform a specific task more consistently.
Pre-trained Model
↓
Training Dataset
↓
Fine-Tuning
↓
Adapted ModelDuring fine-tuning, the model learns from examples related to the target task. The resulting model can become better suited to that specific use case.
Example of Fine-Tuning
Suppose a company wants an AI model to generate customer-support replies in a specific format.
The training data may contain examples such as:
Customer Question
+
Preferred AnswerAfter training on many examples, the model may learn to follow the company's preferred response style more consistently.
Fine-tuning can be useful for:
Specialized tasks.
Consistent response formats.
Specific writing styles.
Classification tasks.
Domain-specific behavior.
Repeated instruction-following tasks.
Prompting provides instructions when the model is being used, while fine-tuning trains the model further using examples.
Prompting | Fine-Tuning |
|---|---|
Gives instructions at request time | Further trains the model |
No additional model training | Requires additional training |
Easy to test and change | Requires suitable training data |
Useful for many general tasks | Useful for consistent specialized behavior |
For example:
Prompting:
"Always answer in this format."
Fine-Tuning:
Train the model using many examples
that follow this format.RAG | Fine-Tuning |
|---|---|
Retrieves information from an external knowledge source | Further trains the model |
Useful for providing current or changing information | Useful for changing model behavior or task performance |
Knowledge remains in the external knowledge base | Learned changes are reflected in the model's parameters |
Good for company documents and frequently changing information | Good for specialized tasks, formats, or response styles |
Information can be updated by changing the knowledge source | Changes may require another fine-tuning process |
Usually does not change model weights | Updates the model's weights |
Yes. RAG and fine-tuning can be used together when an application needs both specialized behavior and external information.
For example:
User Question
↓
RAG Retrieval
↓
Relevant Information
↓
Fine-Tuned Model
↓
Final AnswerThe fine-tuned model can handle the specific task or response style, while RAG provides the latest information needed to answer the question.
Model Serving is the process of making a trained AI model available to applications through an API or other interface.
Once a model is trained, it needs a way to receive requests and return predictions or generated responses. Model serving provides this connection between the AI model and the application.
In simple words, Model Serving makes a trained AI model ready for applications to use.
A basic model-serving flow looks like this:
Application
↓
Model API
↓
Model Server
↓
AI Model
↓
Prediction / Response
↓
ApplicationThe application sends a request to the model server. The server passes the request to the model and returns the result.
Example
Suppose a company has a model that classifies customer messages.
A customer sends:
"I want to cancel my order."The application sends the message to the model server.
Customer Message
↓
Model Server
↓
AI Model
↓
Prediction
↓
ApplicationThe model might classify the message as Order Cancellation.
In production, a model may need to handle:
Many users and requests.
High traffic.
Different input sizes.
Model updates.
Monitoring and logging.
Failures and recovery.
A good model-serving system helps applications use AI models reliably, efficiently, and at scale.
Model Training | Model Serving |
|---|---|
Teaches the model using data | Makes the trained model available for use |
Updates model parameters | Handles application requests |
Usually happens during development | Runs when users or applications need predictions |
Focuses on learning | Focuses on reliable model usage |
When you use an AI chatbot, you may notice that the answer starts appearing before the complete response is ready. This is called a streaming AI response.
In simple words, AI response streaming means showing an AI-generated answer step by step as it is being created. You can start reading the answer without waiting for the complete response.
This makes AI applications feel faster and more interactive.
Let's understand it with a simple example.
Without Streaming
User asks a question
↓
AI generates the complete answer
↓
Application waits
↓
Complete answer appearsWith Streaming
User asks a question
↓
AI starts generating
↓
First part appears
↓
More parts appear
↓
Complete answerWith streaming, the application receives the AI response in small parts and displays them as they arrive.
The main difference is when the user can see the answer.
Normal AI Response | Streaming AI Response |
|---|---|
Waits for the complete answer | Shows the answer as it is generated |
User sees the answer at the end | User can start reading early |
Can feel slow for long answers | Feels more responsive |
Returns one complete response | Receives the response in parts |
For example, if an AI takes several seconds to generate a long answer, streaming lets you start reading while the rest of the answer is still being generated.
Streaming can improve the way users interact with an AI application.
Some key benefits are:
Faster first response: Users can see the beginning of the answer sooner.
Better user experience: Users do not have to stare at a blank screen while waiting.
Useful for long answers: Users can start reading while the AI continues generating.
More natural interaction: The response appears gradually, similar to someone typing.
AI response streaming is commonly used in:
AI Chatbots: Chatbots can display answers as they are generated, making conversations feel more responsive.
Coding Assistants: AI coding tools can show generated code while the response is being created.
AI Writing Tools: Writing tools can gradually display emails, articles, summaries, and other content.
Customer Support: AI support systems can start showing a reply while the remaining response is being generated.
Long AI Responses: Streaming is especially useful when an AI model needs to generate a detailed answer.
Asynchronous AI processing means running an AI task in the background so the user does not have to wait for the task to finish.
It is especially useful when an AI task takes a long time, such as processing a large document, generating a report, or analyzing a large amount of data.
In synchronous processing, the application waits until the AI task is complete before returning the result.
User
↓
Request
↓
AI Processing
↓
Wait
↓
Result
↓
UserThe user has to wait for the task to finish.
In asynchronous processing, the application starts the task in the background and can return a job status while the AI continues working.
User
↓
Request
↓
Create Background Job
↓
Return Job Status
↓
AI Processes in Background
↓
Result Ready
↓
User Gets ResultThis allows the application to handle long-running AI tasks without keeping the user request open until everything is finished.
Example
Suppose a user uploads a 500-page document and asks an AI application to summarize it.
Instead of making the user wait on the same request, the application can create a background job.
Upload Document
↓
Create AI Job
↓
Background Processing
↓
Analyze Document
↓
Create Summary
↓
Notify UserThe user can continue using the application while the AI works in the background.
It is commonly useful for:
Large document processing
Batch AI tasks
Report generation
Image generation
Audio and video processing
Large-scale data analysis
Asynchronous processing is useful when an AI task may take a long time to complete.
It helps applications handle background jobs more efficiently and allows multiple tasks to be processed using queues and worker systems.
In simple words, asynchronous AI processing lets the application start a long AI task without making the user wait for the result immediately.
Building an AI application involves much more than calling an LLM.
The main concepts are:
AI Application Architecture — Defines how the different parts of an AI application work together.
LLM APIs — Allow applications to communicate with language models.
AI Gateway — Provides a common layer for managing AI model requests.
Model Routing — Chooses the right model for a particular request.
Embeddings — Convert information such as text into numerical vectors that can represent meaning.
Vector Databases — Store and search embeddings for similarity-based retrieval.
Semantic Search — Finds information based on meaning instead of only matching exact words.
RAG — Allows an LLM to use information from external knowledge sources when generating answers.
RAG Architecture — Connects document processing, embeddings, retrieval, reranking, and the LLM.
Chunking — Breaks large documents into smaller searchable pieces.
Retrieval — Finds information that is relevant to a user's question.
Reranking — Reorders retrieved results so the most useful information comes first.
RAG vs Fine-Tuning — RAG provides external information, while fine-tuning changes how a model behaves through additional training.
Fine-Tuning — Adapts a pre-trained model for a specific task or behavior.
Model Serving — Makes an AI model available for applications to use.
Streaming AI Responses — Shows the model's response as it is being generated.
Asynchronous AI Processing — Handles longer AI tasks in the background instead of making users wait.