Scribd
Upload
7 views

Chapter 1

Uploaded by

Gunduz Vehbi Demirci
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
7 views

Chapter 1

Uploaded by

Gunduz Vehbi Demirci
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 35

Introducing large

language models
INTRODUCTION TO LLMS IN PYTHON

Iván Palomares Carrascosa, PhD


Senior Data Science & AI Manager
INTRODUCTION TO LLMS IN PYTHON
INTRODUCTION TO LLMS IN PYTHON
INTRODUCTION TO LLMS IN PYTHON
INTRODUCTION TO LLMS IN PYTHON
INTRODUCTION TO LLMS IN PYTHON
LLMs
Based on deep learning architectures
Most commonly transformers

Capture complex patterns in text data

Significant advances in Natural Language


Processing (NLP) tasks

Huge neural networks with lots of


parameters

Trained on enormous text datasets

INTRODUCTION TO LLMS IN PYTHON


LLMs development lifecycle
Data ingestion and preparation
Model architecture design

Pre-training and fine-tuning:

Model evaluation, deployment, monitoring and maintenance

INTRODUCTION TO LLMS IN PYTHON


Pre-training and fine-tuning

Pre-training

Learns patterns

Computationally intensive
Pre-trained foundation model

INTRODUCTION TO LLMS IN PYTHON


Pre-training and fine-tuning

Pre-training Fine-tuning

Learns patterns Domain specific

Computationally intensive

Pre-trained foundation model

INTRODUCTION TO LLMS IN PYTHON


Harnessing pre-trained LLMs: Hugging Face

INTRODUCTION TO LLMS IN PYTHON


Harnessing pre-trained LLMs: Hugging Face

Hugging Face : https://huggingface.co/

A community hub of pre-trained LLMs and datasets

INTRODUCTION TO LLMS IN PYTHON


Using Hugging Face models
from transformers import pipeline

text_classifier = pipeline(task="text-classification",
model="nlptown/bert-base-multilingual-uncased-sentiment")

text = "Dear seller, I got very impressed with the fast delivery and careful packaging

sentiment = text_classifier(text)

print(sentiment)

[{'label': 'POSITIVE', 'score': 0.9998602867126465}]

INTRODUCTION TO LLMS IN PYTHON


Let's practice!
INTRODUCTION TO LLMS IN PYTHON
Tasks LLMs can
perform
INTRODUCTION TO LLMS IN PYTHON

Iván Palomares Carrascosa, PhD


Senior Data Science & AI Manager
Language tasks: overview

INTRODUCTION TO LLMS IN PYTHON


Language tasks: overview

INTRODUCTION TO LLMS IN PYTHON


Language tasks: overview

INTRODUCTION TO LLMS IN PYTHON


Language tasks: overview

INTRODUCTION TO LLMS IN PYTHON


Language tasks: overview

Language task Goal


Code generation Automatically generate code scripts based on requirements
Intent recognition Determine purpose behind a text, e.g., in chatbot enquiries
Named entity recognition Identify and classify named entities in text (people, places, ...)

INTRODUCTION TO LLMS IN PYTHON


Text classification

from transformers import pipeline

llm = pipeline("text-classification")
text = "Walking amid Gion's Machiya wooden houses was a mesmerizing experience"
outputs = llm(text)

print(outputs[0]['label'])

POSITIVE

INTRODUCTION TO LLMS IN PYTHON


Text generation

llm = pipeline("text-generation")
prompt = "The Gion neighborhood in Kyoto is famous for"
outputs = llm(prompt, max_length=100)
print(outputs[0]['generated_text'])

The Gion neighborhood in Kyoto is famous for making fish and seafood by the sea,
which made sense in the 1920s because it was the largest city of its age.

INTRODUCTION TO LLMS IN PYTHON


Text summarization

llm= pipeline("summarization", model="facebook/bart-large-cnn")


long_text = """Walking amid Gion's Machiya wooden houses is a mesmerizing experience. The beautifully
preserved structures exuded an old-world charm that transports visitors back in time, making them feel
like they had stepped into a living museum. The glow of lanterns lining the narrow streets add to the
enchanting ambiance, making each stroll a memorable journey through Japan's rich cultural history."""
outputs = llm(long_text, max_length=60, clean_up_tokenization_spaces=True)
print(outputs[0]['summary_text'])

Walking amid Gion's Machiya wooden houses is a mesmerizing experience. The beautifully preserved
structures exuded an old-world charm. The glow of lanterns lining the narrow streets add to the ambiance.
Each stroll is an memorable journey through Japan's rich cultural history.

INTRODUCTION TO LLMS IN PYTHON


Question-answering

llm = pipeline("question-answering")
context = "Walking amid Gion's Machiya wooden houses was a mesmerizing experience."
question = "What are Machiya houses made of?"
outputs = llm(question=question, context=context)
print(outputs['answer'])

wooden

INTRODUCTION TO LLMS IN PYTHON


Language translation

llm = pipeline("translation_en_to_es", model="Helsinki-NLP/opus-mt-en-es")


text = "Walking amid Gion's Machiya wooden houses was a mesmerizing experience."
outputs = llm(text, clean_up_tokenization_spaces=True)
print(outputs[0]['translation_text'])

Caminar entre las casas de madera Machiya de Gion fue una experiencia fascinante.

INTRODUCTION TO LLMS IN PYTHON


Let's practice!
INTRODUCTION TO LLMS IN PYTHON
The transformer
architecture
INTRODUCTION TO LLMS IN PYTHON

Iván Palomares Carrascosa, PhD


Senior Data Science & AI Manager
What is a transformer?
Deep learning architecture for text processing, understanding, and generation

Characteristics:

No recurrent (RNN) architecture

Capture long-range dependencies in text

Tokens are handled simultaneously

Attention mechanisms + positional encoding

INTRODUCTION TO LLMS IN PYTHON


The original transformer architecture

1 Image source: A. Vaswani, et al. "Attention is all you need". Arxiv, 2017: https://arxiv.org/pdf/1706.03762.pdf

INTRODUCTION TO LLMS IN PYTHON


The original transformer architecture
Two main stacks: encoder and decoder
Each layer: attention mechanisms and
feed-forward computations

Capture complex semantic patterns and


dependencies

No recurrence nor convolutions

Intended for various language tasks, e.g.:


Translation

Summarization

Question-answering

1 Image source: A. Vaswani, et al. "Attention is all you need". Arxiv, 2017: https://arxiv.org/pdf/1706.03762.pdf

INTRODUCTION TO LLMS IN PYTHON


Our first PyTorch transformer
Structural elements: import torch
import torch.nn as nn

d_model : model embedding dimension


d_model = 512

n_heads : number of attention heads n_heads = 8


num_encoder_layers = 6

num_encoder_layers,num_decoder_layers : num_decoder_layers = 6

number of encoder and decoder layers model = nn.Transformer(


d_model=d_model,
PyTorch transformer initialization nhead=n_heads,
num_encoder_layers=num_encoder_layers,

torch.nn.Transformer class num_decoder_layers=num_decoder_layers


)

INTRODUCTION TO LLMS IN PYTHON


Types of transformer architectures

Type Language tasks Notable models


Encoder-decoder Language translation, text summarization T5, BART
-
-

INTRODUCTION TO LLMS IN PYTHON


Types of transformer architectures

Type Language tasks Notable models


Encoder-decoder Language translation, text summarization T5, BART
Encoder-only Text classification, extractive QA BERT
-

INTRODUCTION TO LLMS IN PYTHON


Types of transformer architectures

Type Language tasks Notable models


Encoder-decoder Language translation, text summarization T5, BART
Encoder-only Text classification, extractive QA BERT
Decoder-only Text generation, generative QA GPT

INTRODUCTION TO LLMS IN PYTHON


Let's practice!
INTRODUCTION TO LLMS IN PYTHON

You might also like