BACKGROUND (1/4)

OVERVIEW
 World Wide Web can be viewed as a multimedia database, but
 Introduction: the shoebox metaphor it lacks

 Background: multimedia databases – consistency

Slide 1  Approximate retrieval: use the content of multimedia objects Slide 3 – maintainability

– (more or less) searchability

 Social information filtering
 Most common data object found on WWW:
 New requirements on databases The requested URL /data/onderwijs/studie-

 Research topics gids.1995-1996/˜INF/vakken/214100.html

was not found on this server.

INTRODUCTION BACKGROUND (2/4)

 Problem  Multimedia database

– Dealing with many conventional ‘shoeboxes’ of multimedia data – All traditional database properties
 Storage
– Data types for image, video and audio objects
Slide 2  Retrieval Slide 4
 Sharing – Multimedia objects are first-class citizens

 Solution  Example: commercial system Illustra (Informix)

– Identify generic functionality needed for many different search tasks – OORDBMS - some OO functionality on a RDBMS

– Provide this search functionality using multimedia database systems – Extra functionality provided through datablades
 BACKGROUND (3/4) APPROXIMATE RETRIEVAL (1/5)

create function image display returns void  Content-based retrieval: based on similarity
as external name image.so(display) language C;
– Find all objects that are similar to this object
Slide 5 Slide 7
create type image t( ... ) – Exact similarity finds nothing; need a distance function

– Use representations of the digitized objects that capture some part of

create table images ( image t image, VARCHAR(20) name );
the (syntactic) meaning of the object
insert into images (image, name) values (’arjen.gif’, ’Arjen’);
 Query by Example paradigm
select image display(image) from images where name = ’Arjen’;

BACKGROUND (4/4)
APPROXIMATE RETRIEVAL: TEXT (2/5)

 Database is a tool to retrieve unknown properties using some  Represent full text by its terms
known properties
 Normalized term frequency (tf)
Slide 6
 Problem: how can we express the properties of digitized data Slide 8 – Terms that occur often in a document are representative
objects?

 Manually added descriptions is not a solution  Inverse document frequency (idf)

– Terms that occur in all documents do not add much information
– Different vocabulary of user and system (cf. dark vs. somber)
– Many aspects cannot be expressed unambiguously
 Left and right brain differences?
 Similarity ' tf ( t )
idf ( t
j j )
 QBIC FEATURES
APPROXIMATE RETRIEVAL: SPEECH (3/5)
 Color features
 ‘Conventional’ speech recognition – Color histogram, average color in different color spaces

– Using phonemes, cannot determine word boundaries  Texture features

Slide 9 – Therefore, we need a predefined vocabulary Slide 11 – Contrast, coarseness, directionality
– Not suited for general solution (eg. names)
 Shape features
 Solution: Index Other Features than Words – Area, circularity, eccentricity, axis orientation

 Phoneme sequences V +, V +C +, C + V + and C + V +C +  Sketch features

– Reduced resolution edge map

APPROXIMATE RETRIEVAL: IMAGES (4/5)

 QBIC (Query By Image Content) system APPROXIMATE RETRIEVAL: AUDIO (5/5)

– Database population  Muscle Fish: QBIC for content-based audio retrieval

 Prepare ‘thumbnail’ images
 Assisted sketch outlining – Small amount of reasonable features
Slide 10 Slide 12
– Feature calculation – Query by example paradigm

– Image query
 Iterative querying process
 Supports subjective properties like ‘scratchiness’
 Some features may be suited for direct input
– Each class has a prototype model

 Excellent for retrieving sunset-on-beach pictures

 GEMINI: INTUITION (1/4)

MUSCLE FISH FEATURES (1/2) S1

 Short-time features
Feature2

F(S1)
1 365
e
..
F(Sn)
– Pitch Sn
Feature1
Slide 13 Slide 15
– Loudness
1 365
– Brightness

– Bandwidth  Original data has too high dimensionality

– Harmonicity
 Map S with some F (S ) to f -d feature space
i i

 Find a quick and dirty test in feature space

MUSCLE FISH FEATURES (2/2)

GEMINI: ALGORITHM (2/4)
 Reduce amount of short-time features using
 GEneric Multimedia object INdexIng
– Average

– Variance
 Determine D(O1; O2)
Slide 14 Slide 16
– Autocorrelation  Find Feature Extraction Function
 Prove Dfeature(O1; O2)  D(O1; O2)
– Maximum and minimum

– Parameters expressing shape of smoothed trajectory

 Vector consists of duration plus above parameters  Choose Spatial Indexing Method
 GEMINI: FALSE DISMISSALS (3/4) SOCIAL INFORMATION FILTERING (1/2)

 General intuition
 Mapping Must Preserve Distances – Collect judgements of many people

Dfeature(F (O1); F (O2))  D(O1; O2)

Slide 17 Slide 19 – Use nearest-neighbour algorithms to find similar judgement vectors
k
– Use differences between similar vectors as recommendation

 Proof:  People’s tastes are not randomly distributed

"
D(Q; O)  " ) Dfeature(F (Q); F (O))  k
 Commercialized for film and music in firefly system

GEMINI: TIME SERIES EXAMPLE (4/4)

SOCIAL INFORMATION FILTERING (2/2)

 Euclidian Distance  Benefits over content-based-filtering approach

vuuX
t
n – Overcomes problems with identification of suitable features for objects
Slide 18 D(S; Q) (Si Qi )2 Slide 20 like music and art
=1
i
– Inherent method for serendipitous finds

 Discrete Fourier Transform – Deals implicitely with qualitative aspects like style

 Parseval’s Theorem  Large groups, broader domains?

 R-trees
 NEW REQUIREMENTS ON DATABASES

 Capability to process queries spanning multiple media INQUERY (2/2)

– The caption ‘Kok’ and the picture can only together resolve the  Pros
question whether we search the Dutch minister-president or the
muppet show. – Very good full-text retrieval system

 Querying is an interaction process

Slide 21 Slide 23 – Allows application to other data types

– Query by example (however, QBE alone is not sufficient)

 Cons
– Relevance feedback – Unknown performance on imprecise data

 Query processing must incoorporate social information filtering – Not a nice document model

techniques

INQUERY (1/2) FURTHER WORK

 Investigate Bayesian inference networks in database

architecture

– Combination of evidence from different ‘agents’

Slide 22 Slide 24 – Allows integration of knowledge about erroneous recognition

– Execution performance?

 Text retrieval system with combined evidence and relevance  Investigate latent semantic indexing
feedback
 Investigate what is needed for a ‘return on investment’ analysis
 Estimate P (I j document) using Bayesian inference networks of recognition agents
 APPROACH: THE FIRST STEPS

 Research applicability of Bayesian framework

Slide 25
 Master’s project: phoneme sequence recognizer
 Master’s project: sub-pattern matching
 Design project: intelligent television
 ...