Durgesh Tiwari
Author
Artificial Intelligence (AI) is technology that enables computer systems to perform tasks that normally require human intelligence.
These tasks include:
Understanding language.
Recognizing images.
Making decisions.
Solving problems.
Learning from data.
Generating content.
In simple words, AI enables machines to perform intelligent tasks.
AI can automate repetitive tasks, process large amounts of data, improve decision-making, and help solve complex problems.
A basic AI system can be understood through this process:
Data
↓
AI Model
↓
Learning / Processing
↓
Prediction or Decision
↓
OutputThe AI model processes data, identifies patterns, and uses what it has learned to produce an output.
AI is commonly used in:
Chatbots and virtual assistants.
Recommendation systems.
Image and speech recognition.
Fraud detection.
Autonomous systems.
Healthcare applications.
Search engines.
Machine Learning (ML) is a subset of Artificial Intelligence (AI) in which systems learn patterns from data and use them to make predictions or decisions.
In simple terms, Machine Learning allows computers to learn from data rather than being programmed for every situation.
Deep Learning is a subset of Machine Learning that uses multi-layer neural networks to learn complex patterns from large amounts of data.
Deep Learning is commonly used for:
Image recognition.
Speech recognition.
Natural language processing.
Generative AI.
The relationship between AI, Machine Learning, and Deep Learning can be understood as:
Artificial Intelligence
│
└── Machine Learning
│
└── Deep LearningAI → The broad field of building intelligent systems.
ML → Systems learn patterns from data.
Deep Learning → Uses multi-layer neural networks to learn complex patterns.
Consider a spam email detection system:
AI → The overall system used to detect spam emails.
ML → Learns patterns from previous spam and non-spam emails.
Deep Learning → Uses neural networks to identify complex patterns in email content.
Generative AI is a type of AI that creates new content by learning patterns from training data.
It can generate:
Text.
Images.
Audio.
Video.
Code.
In simple words, Generative AI uses learned patterns to create new content.
A simplified Generative AI process looks like this:
Training Data
↓
AI Model
↓
Learns Patterns
↓
User Prompt
↓
Generated ContentThe model learns patterns from training data and uses them to generate content based on the user's prompt.
Text Generation: Generates human-like text such as articles, emails, summaries, and answers.
It is commonly used in chatbots, content creation, and AI assistants.
Image Generation: Creates new images from text prompts or other inputs.
It can be used for illustrations, product designs, marketing content, and creative artwork.
Code Generation: Generates programming code based on natural-language instructions.
It can help developers write functions, fix bugs, explain code, and create applications.
Audio Generation: Generates speech, music, sound effects, and other types of audio.
It is used in voice assistants, audiobooks, podcasts, and music creation.
Video Generation: Creates videos from text prompts, images, or existing content.
It can be used for advertisements, educational videos, animations, and other visual content.
Feature | Traditional AI | Generative AI |
|---|---|---|
Main Purpose | Makes predictions or classifications | Creates new content |
How It Works | Uses learned patterns to make decisions | Uses learned patterns to generate content |
Example | Spam detection | Text generation |
Output | Prediction, classification, or decision | Text, images, code, audio, or video |
Generative AI is commonly used for:
AI assistants.
Content generation.
Code generation.
Document summarization.
Creative applications.
Customer support.
A Large Language Model (LLM) is an AI model trained on large amounts of text data to understand and generate human language.
LLMs can perform tasks such as:
Answering questions.
Summarizing text.
Generating content.
Translating languages.
Writing code.
Extracting information.
In simple words, an LLM is an AI model that can understand and generate language.
LLMs can perform many language-related tasks using natural-language instructions. Users can describe what they need in normal language instead of using a separate program for every task.
An LLM processes the input and predicts the next likely token based on the available context.
User Prompt
↓
Tokenization
↓
LLM
↓
Next-Token Prediction
↓
Generated ResponseLLMs go through different stages during their development and use.
During training, the model learns language patterns from large datasets and adjusts its parameters based on what it learns.
Training Data
↓
Model Training
↓
Learned Parameters
↓
Trained ModelPre-training is the initial training stage where the model learns general language patterns from large datasets.
It helps the model learn relationships between tokens and develop general language capabilities.
Fine-tuning further trains a pre-trained model on a more specific dataset or task.
It can adapt the model for particular use cases or improve its behavior for specific tasks.
Inference is the process of using a trained LLM to generate an output from a given input.
Prompt
↓
Trained Model
↓
Inference
↓
ResponseLLMs generate text by predicting one token at a time based on the input context and learned parameters. This process continues until the response is complete.
Examples of LLMs
Examples of LLMs include models developed by organizations such as:
OpenAI.
Google.
Meta.
Anthropic.
Mistral.
LLM Architecture describes the internal structure and components that allow a Large Language Model (LLM) to process input and generate output.
Most modern LLMs are based on the Transformer architecture.
The Transformer is a neural network architecture that processes relationships between tokens using mechanisms such as attention.
A simplified LLM flow looks like this:
Input Text
↓
Tokenization
↓
Token Representations
↓
Transformer Layers
↓
Output Probabilities
↓
Generated TokensThe Transformer processes the input through multiple layers and produces representations that the model uses to predict the next tokens.
Transformer-based models can use different architectural designs:
Encoder — Processes input and creates useful representations.
Decoder — Generates output tokens based on the available context.
Encoder-Decoder — Uses both components to process an input and generate a related output.
Different models use these components in different ways. Many modern LLMs, especially text-generation models, use decoder-based Transformer architectures.
The Attention Mechanism helps the model determine which parts of the input are important when processing each token.
For example, when processing a sentence, attention helps the model focus on relevant words to better understand their relationship and context.
Self-Attention allows tokens in the same sequence to consider their relationships with other tokens.
This helps the model understand how different words relate to each other within the given context.
Feed-Forward Networks are another important part of Transformer layers.
They further process the representations produced by the attention mechanism before passing them to the next layer.
A token is a unit of text that an LLM processes.
A token can represent:
A complete word.
Part of a word.
A punctuation mark.
Another piece of text.
Tokens are not always the same as words.
Tokenization is the process of breaking text into tokens that an LLM can process.
"Hello world!"
↓
Tokenizer
↓
TokensThe tokenizer breaks the input text into model-specific tokens and converts them into numerical representations that the model can process.
A single word can be represented by:
One token.
Multiple tokens.
Therefore, token count is not always equal to word count.
Tokenization affects:
Input size.
Context window usage.
Processing cost.
Model performance.
The amount of text a model can process.
The Context Window is the amount of tokenized information an LLM can process and consider during a particular interaction or model operation.
It can include:
User prompts.
Previous conversation messages.
System instructions.
Other provided content.
A simplified flow is:
System Instructions
+
Conversation
+
User Prompt
↓
Context Window
↓
LLM
↓
ResponseThe LLM uses the information available in the context window to understand the current request and generate a response.
Context length is usually measured in tokens.
Different LLMs support different context-window sizes. A larger context window allows a model to process more information in a single interaction.
Every model has a maximum context limit.
If the input and generated content exceed this limit, some information may need to be shortened, removed, or otherwise managed before the model can continue processing it.
Context Window | Memory |
|---|---|
Holds information available during a particular interaction | Can retain information across interactions |
Limited by the model's context size | Managed through a separate memory mechanism |
Used to process the current context | Used to make retained information available later |
Includes information provided in the current interaction | May contain information saved from previous interactions |
In simple words, the Context Window contains information the model can use in the current context, while Memory is a separate mechanism that can retain information across interactions.
Model Parameters are numerical values that an AI model learns during training.
They help the model recognize patterns in data and determine how it processes input to produce an output.
In simple words, parameters are learned values that help an AI model use patterns learned from data.
During training, the model makes predictions and compares them with the expected results. Based on the error, the model updates its parameters to improve future predictions.
Training Data
↓
Prediction
↓
Error
↓
Parameter Updates
↓
Improved ModelThis process is repeated many times until the model learns useful patterns from the training data.
A model with more parameters has a larger number of learned values. However, more parameters do not automatically mean better performance.
Model performance also depends on factors such as:
Model architecture.
Training data.
Training methods.
Model quality.
Parameters are a fundamental part of a trained AI model. They allow the model to represent and use patterns learned during training.
Inference is the process of using a trained AI model to produce an output from an input.
For an LLM, a simplified inference process is:
Prompt
↓
Tokenization
↓
Model Processing
↓
Token Prediction
↓
Generated ResponseDuring inference, the LLM processes the input and available context, then predicts the next token.
The model generates tokens one by one until it reaches an appropriate stopping condition or the configured output limit.
Training | Inference |
|---|---|
Model learns patterns from data. | Trained model uses learned patterns to generate an output. |
Parameters are updated during the learning process. | Parameters are used without being updated during normal inference. |
Data → Learning → Trained Model | Prompt → Trained Model → Output |
The speed and cost of LLM inference can be affected by:
Model size: Larger models generally require more computing resources.
Input length: Longer inputs can increase processing time and cost.
Output length: Longer responses require more computation and tokens.
Hardware: More capable hardware can improve inference speed.
Number of users: More concurrent users can increase resource requirements.
Serving architecture: The way a model is deployed and served can affect latency, throughput, and resource usage.
Common metrics used to evaluate inference performance include:
Latency — How long the model takes to produce a response.
Throughput — How many requests or tokens can be processed over a period of time.
Memory Requirements — How much memory is needed to run the model.
Compute Requirements — How much processing power the model needs.
Context Size — How much input information the model needs to process.
These factors are important when deploying LLMs in real-world applications.
Prompt Engineering is the practice of writing clear and specific instructions to get useful and reliable results from AI models.
In simple words, Prompt Engineering means giving an AI model clear instructions about what you want it to do.
A well-written prompt helps an AI model understand:
The task.
The required context.
The expected output.
The level of detail.
Any specific constraints.
Clear prompts can make responses more relevant and consistent.
A useful prompt can include:
Role
+
Task
+
Context
+
Constraints
+
Output FormatFor example, instead of asking:
Explain Kubernetes.You can provide more specific instructions:
Explain Kubernetes in simple English
for a beginner and include a short example.The second prompt gives the model clearer instructions about the audience, language, and expected response.
Few-shot prompting provides examples to show the model the expected input-output pattern.
Example 1 → Output
Example 2 → Output
New Input → ?The examples help guide the model toward the desired format or behavior.
Chain-of-thought prompting refers to techniques designed to encourage structured reasoning when solving complex problems.
For practical use, it is generally better to ask for a concise explanation, key steps, or final answer rather than requesting hidden internal reasoning.
Role-based prompting gives the AI model a specific role or perspective.
Example:
You are a technical interviewer.
Ask me backend development questions.This gives the model additional context about how it should approach the task.
Prompt optimization is the process of improving prompts through testing and iteration.
Teams can compare different prompts and refine them to produce more consistent, accurate, and useful results.
A System Prompt provides high-level instructions that guide how an AI model should behave during an interaction.
It can define:
Role.
Behavior.
Rules.
Response style.
Constraints.
A system prompt provides general instructions before the model handles a user's specific request.
These instructions can guide the model's behavior, response style, task boundaries, and other requirements.
System Prompt | User Prompt |
|---|---|
Provides high-level instructions | Provides the user's request |
Defines behavior and constraints | Defines the specific task |
Guides the overall interaction | Usually focuses on the current request |
System instructions can influence:
Response style.
Task boundaries.
Allowed behavior.
Output requirements.
Rules the model should follow.
A good system prompt should be:
Clear — Easy to understand.
Specific — Clearly defines what the model should do.
Consistent — Avoids conflicting instructions.
Unambiguous — Leaves little room for confusion.
Relevant — Focuses on requirements that matter for the task.
Model Selection is the process of choosing the right AI or LLM model for a specific application.
Different models can vary in their capabilities, cost, context window, speed, and overall performance.
Important factors include:
The model should perform well on the tasks required by the application.
For example, a coding application may need a model with strong programming capabilities, while a customer-support application may prioritize strong language understanding and reliable responses.
Applications that work with long documents or large amounts of information may need a model with a larger context window.
LLM usage can have different costs depending on factors such as input and output token usage.
Teams should consider both the cost per request and the expected usage of the application.
Latency is the time a model takes to produce a response.
Applications that require fast responses may prefer models or infrastructure that provide lower latency.
Model performance should be evaluated using relevant benchmarks and, when possible, real-world tests based on the application's actual requirements.
A model that performs well on a general benchmark may not always be the best choice for a specific application.
A model should provide consistent results and remain suitable for the application's expected workload.
Reliability can be evaluated through testing, monitoring, and real-world usage.
Model selection also depends on how the model will be deployed and what data it will process.
For example, some applications may require self-hosting or stronger control over data and infrastructure, while others may prefer a managed API.
Open-source models are models whose availability and licensing allow users to access and use their components according to the specific license.
Depending on the model and license, users may be able to:
Access model weights.
Inspect the implementation.
Run the model themselves.
Customize or adapt the model.
Important: In AI, the term "open-source" can mean different things depending on the license and which components are actually released.
More control over deployment.
Greater customization potential.
Can support self-hosting.
More control over infrastructure and data.
Requires infrastructure and technical management.
Hardware costs can be significant.
Performance and licensing vary by model.
Closed-source models are generally provided through controlled access, such as an API or hosted service. Their internal implementation or model weights are not fully available to users.
Easy access through hosted services or APIs.
No need to manage the model infrastructure directly.
Managed scaling and updates.
Less control over the underlying model.
Dependency on the provider.
Usage costs and service limitations may apply.
Open-Source Models | Closed-Source Models |
|---|---|
More control over deployment | Provider manages deployment |
Can support self-hosting | Usually accessed through a hosted service or API |
Greater customization potential | Less control over the underlying model |
Infrastructure may be required | Less infrastructure management |
Licensing varies by model | Provider terms apply |
More control over infrastructure and data | Data handling depends on the provider and service |
Open-source models can be suitable when an organization needs greater control, customization, self-hosting, or infrastructure flexibility.
Closed-source models can be suitable when an organization wants easy access to AI capabilities without managing the underlying model infrastructure.
In simple words, open-source models can provide more control and customization, while closed-source models usually provide easier access with less infrastructure management.
AI enables machines to perform tasks that normally require human intelligence.
Machine Learning allows systems to learn patterns from data and use them for predictions or decisions.
Deep Learning uses multi-layer neural networks to learn complex patterns from data.
Generative AI creates new content such as text, images, audio, video, and code.
LLMs are AI models designed to understand and generate human language.
Transformers are the foundation of many modern LLM architectures.
Tokens are units of text processed by language models.
Context Windows define how much tokenized information an LLM can process as context.
Model Parameters are learned numerical values that help a model represent and use patterns learned from data.
Inference is the process of using a trained model to generate an output.
Prompt Engineering helps users create clear and effective instructions for AI models.
System Prompts provide high-level instructions that guide AI model behavior.
Model Selection involves choosing a model based on quality, context window, cost, latency, and application requirements.
Open-Source and Closed-Source Models provide different trade-offs in control, customization, infrastructure, privacy, and ease of use.