Organizing Multimedia

1
Organizing Multimedia

• Issues, Standards, Examples

2
Definitions of Multimedia

• General Definition
• The presentation of text, graphics, video,
  animation, and sound in an integrated way. 1
• Interactive Multimedia
• text, graphics, video, animation, and sound that
  can be manipulated by the viewer. 2

3
Multimedia What for?

• Information
• Training
• Meetings
• Conference calls
• Maps
• Teaching/learning

• Entertainment
• Movies
• Music
• Audio programs
• Documentaries
• Games

• Augmentation
• information services
• sensor data
• Personalized multimedia

4
Issues for Organizers

• Describing content
• Semantic description
• Accessing specific content
• Segmentation
• Standards
• Audio
• Video
• metadata

5
Categories of multimedia

• Audio
• Video/Film
• Online/Interactive
• Games

6
Issues

• The biggest issue is bandwidth!
• Video
• compression
• frame rate
• Dimensions
• Audio
• compression

7
Ways to segment Video

• text processing (technical coding processing)
• image processing
• -color histogram
• -optical flow
• speech processing

8
Extracting Information

• Histograms

9
Segmenting Optical Flow
10
Ways to segment Audio

• -events silence, music, explosions, applause
• -speech high energy vowels, low energy
  consonants
• Semantic segmentation
• Who, what, where

srinivasan
11
Solutions

• traditional methods of organization using
  categories, descriptions
• not rich
• human efforts
• expensive
• Automatic
• Inaccurate
• Difficult to extract semantic meaning

12
MPEG-7 Overview

• MPEG Family
• MPEG-1, -2, and -4
• MPEG-7and Its Objective
• Main Elements of MPEG-7
• Scope of MPEG-7
• Application Examples

13
MPEG Family

• Moving Pictures Experts Groups
• Defining standards for the coding of video and
  audio information
• Billions of USD

14
MPEG-1, -2, and -4

• MPEG-1 aim to record movies of VHS quality with
  CD audio quality on CD-ROM
• MPEG-2 include coding for different applications
• MPEG-4 the 1st technology platform designed for
  network-based multimedia delivery (Grossman,
  2000)

15
MPEG-7 and Its Objective

• Multimedia Content Description Interface
• Objective standardize content-based description
  for various types of audiovisual information,
  allowing quick and efficient content
  identification, and addressing a large range of
  applications
• MPEG-1, -2, -4---the bits MPEG-7---the bits
  about the bits.

16
Main Challenge of MPEG-7

• MPEG-7 addresses many different applications in
  many different environments.
• It needs to provide a flexible and extensible
  framework for describing audiovisual data.

17
Main Elements of MPEG-7

• Description Tools
• Descriptors (D)
• Description Schemes (DS)
• A Description Definition Language (DDL)
• System Tools

18
MPEG-7 Main Elements
19
Scope of MPEG-7
20
Application Examples

• Architecture, real estate, and interior design
  (e.g., searching for ideas).
• Broadcast media selection (e.g., radio channel,
  TV channel).
• Cultural services (history museums, art
  galleries, etc.).
• Digital libraries (e.g., image catalogue, musical
  dictionary, bio-medical imaging catalogues, film,
  video and radio archives).
• E-Commerce (e.g., personalized advertising,
  on-line catalogues, directories of e-shops).
• Education (e.g., repositories of multimedia
  courses, multimedia search for support material).
  
• Home Entertainment (e.g., systems for the
  management of personal multimedia collections,
  including manipulation of content, e.g. home
  video editing, searching a game, karaoke).

21
Application Examples (contd)

• Investigation services (e.g., human
  characteristics recognition, forensics).
• Journalism (e.g. searching speeches of a certain
  politician using his name, his voice or his
  face).
• Multimedia directory services (e.g. yellow pages,
  Tourist information, Geographical information
  systems).
• Multimedia editing (e.g., personalized electronic
  news service, media authoring).
• Remote sensing (e.g., cartography, ecology,
  natural resources management).
• Shopping (e.g., searching for clothes that you
  like).
• Social (e.g. dating services).
• Surveillance (e.g., traffic control, surface
  transportation, non-destructive testing in
  hostile environments).

22
VoiceXML Overview

• What is VoiceXML
• Key Concepts of VoiceXML
• VoiceXML Architecture Model
• VoiceXML Gateway Within a Network Infrastructure
• A Simple Sample
• Application Areas

23
What is VoiceXML

• a standard based on XML that allows web
  applications and content to be accessed by a
  phone

24
Key Concepts of VoiceXML

• Sessions
• Dialogs
• Forms
• Menus
• Applications
• Grammars
• Events

25
VoiceXML Architecture Model
26
VoiceXML Gateway Within a Network Infrastructure
27
A Simple Example

• Say one of
• rt.vxml"
• Sports
• tro.vxml"
• Weather
• vxml"
• News
• Please say one of

28
Example (contd)

• Computer Say one of Sports Weather News.
• Human Astrology
• Computer I did not understand what you said.(a
  platform-specific default message.)
• Computer Say one of Sports Weather News.
• Human Sports
• Computer (proceeds to http//www.sports.example/s
  tart.vxml)

29
Application Areas

• Voice portals
• Location-based services
• Voice alerts (such as for advertising)
• Commerce

30
Current Work

• Informedia
• Computer Science
• Carnegie Mellon University
• Open Video Digital Library
• Information Library Science
• UNC-Chapel Hill

31
Informedia

• Part of NSF Digital Libraries Initiative
• Two Phases
• Informedia I
• Started 1994
• Howard Wactlar (http//www-2.cs.cmu.edu/hdw/)
• Informedia II
• Continuation of Informedia I
• Incorporate new technologies

32
Informedia I

• Focus on multimedia storage, search, and
  retrieval
• Began with Public Domain materials
• Partnered with Private Sector businesses
• Developed mechanisms for pay-per-view accounting
  and reduced cost billing systems

33
Informedia IMethodology

• Use Speech Recognition software to transcribe
  audio part of each file
• Store transcribed text in a database and
  associate with a time stamp
• Retrieve video paragraphs based on search terms
• Allow for video skimming of retrieved videos

34
Informedia II

• Added TV and Radio news files and Documentary
  footage
• Setup up automated content capture system
• Incorporates image recognition and natural
  language processing
• Individual files are segmented into video
  paragraphs as they are added to the collection

35
Informedia II

• Returns summaries or abstractions from several
  video paragraphs rather then particular documents
• Allows for multimodal searching

36
Informedia II Interface
37
Open Video Project

• Began at University of Maryland in 1996
• Management of the Project moved to SILS in 1998
• Dr. Marchionini and Dr. Wildemuth
• Creating a open source digitized video repository
  for research purposes
• Primarily digital video, multimedia retrieval,
  and digital library research

38
Open Video Project

• Receives content contributions from other
  multimedia research projects and researchers
• Currently 1,800 video files. 460 hrs of video
• Intended to be used as test collection to compare
  multimedia retrieval systems
• Similar to TREC conferences for text retrieval
• Also intended to be used for research on
  interfaces for video browsing, searching,
  evaluating, and previewing.

39
Open Video Project

• Videos segmented
• Mostly manually
• Uses Dublin Core Compliant Metadata
• Also mostly a manual process
• Work in progress to make more automatic
• Uses a variety of multimedia formats
• MPEG-1, MPEG-2, MPEG-4 and QuickTime

40
Open Video project

• Browse/Search Interface
• Results Interface
• Segment Details

41

• Questions?

42
What is video

• Dialog
• Images
• narration
• Other soundtracks

43
What is Audio

• pitch
• loudness
• duration

44

• As networked multimedia continues to expand into
  business, personal, and societal spaces, the need
  to properly understand the issues and solutions
  for creating and organizing multimedia becomes
  vital to the future of human knowledge.
• Without accurate description or access, much of
  the multimedia information will be ephemeral.