Mechatronics - PowerPoint PPT Presentation

1 / 40
About This Presentation



Vision: probably the most common. Touch: force sensors, strain gauges. Sound: proximity sensors ... Most common type of camera used in machine vision ... – PowerPoint PPT presentation

Number of Views:3424
Avg rating:3.0/5.0
Slides: 41
Provided by: profdenz


Transcript and Presenter's Notes

Title: Mechatronics

Mechatronics RoboticsLecture 11 External
sensors machine vision
  • Dr Vesna Brujic-Okretic
  • Room 11cBC03 or
  • MSRR Lab.
  • Ext. 9676 or 9681
  • Email

Week 11 External sensors and intelligent
robots1) Sensors for robotics 2) Machine
vision3) Image processing4) Applications of
machine vision5) Implementation issues
Tutorial questions
  • External Sensors for robotics
  • Robots are highly flexible in the sense that most
    controllers are fairly easily reprogrammed to
    cope with deliberate variations in the task but
    minimally adaptable in the sense of coping with
    accidental variations in an imperfect world.
  • We attempt to get around this by using external
  • Vision probably the most common
  • Touch force sensors, strain gauges
  • Sound proximity sensors
  • Smell not usually
  • Taste not likely

  • Machine vision - giving robots sight
  • Potentially the most powerful sensor is
    artificial vision
  • Also called computer vision and machine vision.
  • Machine vision is already used for both robotic
    non-robotic applications.
  • Machine vision is a complex subject encompassing
    a number of different high tech fields including
    optical engineering, analogue video processing,
    digital image processing, pattern recognition and
    artificial intelligence and computer graphics.
  • The basic process is modelled on the mammalian
    eye/brain interaction.

  • The mammalian visual process
  • Vision is the process of converting sensory
    information into the knowledge of shape, identity
    or configuration of objects.
  • Other sensors besides light sensors can also
    provide similar information including bat
    sonar, and touch.
  • Previous input and its interpretation can greatly
    affect current processing of sensory data.
  • Seeing is the physical recording of the pattern
    of light energy received from the environment. It
    consists of
  • selective gathering in of light
  • projection or focusing of light on a
    photoreceptive surface
  • conversion of light energy into chemical or
    electrical activity
  • Information from sensors is usually not just ON
    or OFF, but also includes ''how much''.

  • The human vision system
  • Incident light falls on the receptive part of the
  • This is then converted into tiny electrical
    signals which are sent via the visual cortex to
    the brain
  • There is some evidence to suggest that the visual
    data is compressed by neurones in the visual
    cortex before it is processed since there are
    very much fewer dendrites (connections) in the
    early part of the visual cortex than there are
    receptive elements in the retina
  • Humans have two eyes which gives them the ability
    to perceive depth, as first discussed by
    Descartes in the 17th Century.

  • Components of a machine vision system
  • Main elements camera,
  • digitiser, frame-store, processor.

  • Lighting
  • Good illumination of the subject is essential for
    successful application of computer vision
  • Poor image quality cannot be rectified by further
    stages of image processing
  • controlled lighting ensures consistent data from
    the image.
  • There are several methods of illumination
  • front lighting is used to determine surface
    characteristics of the object,
  • back lighting is used for simple outline
  • Structured lighting is used to recover 3D
    geometry of a subject and
  • strobe lighting may be used to freeze a scene
    such as objects on a moving conveyor belt.

  • Camera types
  • The most common input devices for computer vision
    systems are either vidicon or solid state (CCD)
  • (1) TV or Vidicon camera
  • The operating principle of the vidicon camera is
    based upon the image pick-up tube often found in
    television cameras.
  • Image focused onto a photo-conductive target.
  • Target scanned line by line horizontally by an
    electron beam
  • Electric current produces as the beam passes over
  • Current proportional to the intensity of light at
    each point.
  • Tap current to give a video signal
  • Limited resolution finite number of scan lines
    (about 625) and frame rate (30 or 60 frames per
  • Unwanted persistence between one frame and the
  • Non-linear video output with respect to light
  • suffer from blooming of the image around very
    bright light spots, geometrical distortion of the
    image across the photoconductor and sensitivity
    to environmental conditions

  • Camera types
  • (2) CCD camera
  • CCD cameras are made up from several
    photodetectors which may either be arranged in a
    line or in an array.
  • Most common type of camera used in machine vision
  • In its basic form it is a single IC device
    consisting of an array of photosensitive cells
  • Each cell produces an electric current dependent
    on the incident light falling on it.
  • These cells are polled to produce a video signal
    output. In the UK standard colour (PAL) and b/w
    (CCIR) video is 25Hz.
  • CCD cameras have less geometric distortion and a
    more linear video output when compared to
  • since these devices incorporate discrete elements
    they are much better suited to digital processing
  • The large surveillance market means they are
    cheap, robust and easy to use and integrate with
    other systems

  • CCD camera types
  • older systems were very chunky, heavy power
  • advances in IC technology have resulted in
    smaller and smaller systems that use less power
  • miniaturisation first resulted in remote head
    lipstick sized cameras
  • latest technologies include very small single
    board systems

remote head camera
typical camera
  • Image digitiser and display module
  • Digitisation/display is achieved by A/D and D/A
    converters respectively.
  • In older systems both functions are normally
    implemented on a single computer interface card -
    typically architectures include ISA, VME.
  • Newer PCI cards usually send video to the
    graphics adapter - either via the PCI bus or by a
    direct digital video bus.

  • Image digitisation
  • digitisers consist of an analogue amplifier and
    signal conditioner, which give gain and offset
    control on the incoming image data, and also
    provide synchronisation for the signal.
  • Most are capable of converting input analogue
    signals into digital values at full video rates
    around 1/30 - 1/60th second per image.
  • Most digitisers have very limited spatial
    resolution (since standard video is also limited
    in this fashion) - typical output image sizes are
    640x480, 512x512.
  • The digitiser is typically provided with lookup
    tables (LUTs) to quickly change pixel values. For
    each possible pixel value the corresponding
    output value is 'looked up' in the table.
  • Most b/w digitisers are 8 bit. Colour ones
    usually 24bit.
  • The output images from the A/D are usually fed to
    an onboard FRAMESTORE.

  • Image digitisation

  • Framestore module
  • A framestore issimply RAM.
  • In older computers system RAM wasvery limited
    and could not storethe amounts of data required
    forimaging applications .
  • Nowadays systems frequently only have one or two
    times as much RAM as the size of the image they
    are digitising - unless they have an onboard
  • A 640x480 RGB colour picture obviously requires
    (640x480x3) bytes of storage space. Colour images
    can be stored as three channels of red blue and
    green or in other formats.
  • It should be made clear that computer vision
    requires a considerable amount of computer
    memory, even at minimum resolutions

  • Image processing systems
  • Once an image is held in framestore it can be
    frozen and processed
  • A processor module must have access to the
    framestore memory
  • Pipeline image processors have a separate
    hardware device connected to the framestore via a
    dedicated image bus - thus no transfers of image
    data are necessary over the host bus
  • Co-processor systems have a chip onboard the
    digitiser/framestore module which has direct
    hardware access to the framestore - again no
    transfers of image data are necessary over the
    host bus
  • Some systems have daughter-board co-processors
    which may be linked via an image bus - normally a
    mere ribbon cable
  • Many newer, cheaper, systems have no facility for
    pipeline processing or co-processors and expect
    that the host system will do the image processing
    - this requires image transfers over the host
    bus, and a very fast CPU

  • Pipeline image processing
  • Separate hardware modules for each function
  • Each module is interconnected via a digital video
  • Host computer is only used for programming.
  • Very high rates (frame rates) of processing are
    possible. RISC processor Reduced Instruction Set
  • Expensive, difficult to program

digital video bus
system bus
  • Co-processor systems
  • Single board solution - very flexible
  • Co-processor may be mounted on a daughter-card
  • Extra co-processors may be added for a scaleable
  • Almost real time performance can be achieved
  • Programming is easier than pipelined solutions
    through use of dedicated black-box libraries is
    normally necessary

digitiser/ display/framestore/co-processor
system bus
  • Image processing with the system processor
  • Inexpensive - only extra item is the
  • Software development is all targeted at the host
    - relatively simple and can be done using
    libraries, C, or even Java.
  • Speed dictated by the system - Windows operating
    systems can slow the system down unpredictably
  • Display can be a problem - dictated by
    performance of graphics card

  • Image processing stages
  • Once an image hasbeen captured we needto make
    some sense of it
  • Image processing involves three main actions
  • image enhancement,
  • image analysis and
  • image understanding, as shown in Figure.
  • Image processing canbe a complex and difficult
    operation andis still very much aresearch field

  • Image enhancement
  • GOAL
  • to remove unwanted artefacts such as noise,
    distortion, and non-uniform illumination from the
  • local area averaging (mean, mode, median
  • image warping, image subtraction
  • background flattening, contrast enhancement,
    histogram equalisation
  • operations are typically applied from a library
    and are not generally very application specific
  • many low level operations such as filtering can
    be done in real time (I.e. at frame rates) and in

  • The local area analysis of an image

  • Image analysis
  • converts the input image data into a description
    of the scene
  • GOAL
  • to identify and describe, in objective terms,
    the objects in a given image
  • Techniques image segmentation, regions
    labelling, further processing
  • thresholding, edge detection, object labelling
  • blob and shape analysis, perimeter coding,
  • extraction of syntactic or contextual
    descriptions, point matching
  • operations are typically very application
  • most operations are based on a generic theory
    but require large amounts of programming to work
    with a specific application

  • Image understanding
  • Input a description of the image
  • GOAL
  • classify each object and attempt to generate a
    logical decision based on the content of the
    image (e.g. the red object is at location
    x,y,z, or reject the component, or this is
    not a sheep, or there is an intruder").
  • pattern recognition, pattern matching
  • use of knowledge based systems and neural
  • Possible Outcomes further info required, objects
    not recognised, objects successfully recognised
  • operations are completely application specific

  • Example of image processing robotic fishing
  • our goal is to detect, classify and intercept
    cardboard fish as they are presented to the
    robot on a pallet
  • the fish can be one of three species and can be
    presented in any orientation
  • the camera and robot coordinate systems must
    first be calibrated
  • the calibration requires at least a translation,
    a rotation and a scaling - how do we do it ?

  • Raw image pre-processing (inversion)

raw image
inverted image
  • Image segmentation (global thresholding)

threshold too low (128)
threshold too high (220)
  • Image analysis/object labelling

threshold applied at 180
objects detected
  • Object analysis -

Object (003) Object
Co-ordinates (137,275) Transverse
length (pixels) 98 Longitudinal length
(pixels) 399 Extent of object (pixels)
12758 Mean grey level 37
Elongation ratio 97.6216 Object
(004) Object Co-ordinates
(286,369) Transverse length (pixels)
154 Longitudinal length (pixels) 248 Extent
of object (pixels) 24786 Mean grey level
33 Elongation ratio
Object (001) Object
Co-ordinates (409,223) Transverse
length (pixels) 83 Longitudinal length
(pixels) 401 Extent of object (pixels)
13407 Mean grey level 34
Elongation ratio 91.8998 Object
(002) Object Co-ordinates
(235,123) Transverse length (pixels)
197 Longitudinal length (pixels) 175 Extent
of object (pixels) 19365 Mean grey level
37 Elongation ratio
  • Example of pattern recognition how could we use
    a neural network to classify our fish ?
  • a neural network generally requires training -
    that is it is shown a great number of example
    inputs and, for each case, its required output
  • during the training stage it constantly
    restructures its internal state (the weightings
    between its interconnections)
  • once trained it should be able to generalise
    about new fish shapes that it is shown
  • the main problem (or advantage !) with neural
    networks is that they are a black box solution
    to often difficult problems

  • Example of machine vision in manufacturing
  • seam tracking in robotic welding
  • when arc welding of sheet metal plates it is not
    always possible to accurately program the
    desired weld path due to variations in the sheet
  • an on-line method must be used to accurately
    feedback the required trajectory to the robot
  • a laser stripe is used to illuminate the joint
    profile (usually a V)
  • it is then possible to detect the centre of the
    joint using image processing

  • Example of machine vision in manufacturing
  • seam tracking in robotic welding
  • only a single starting point needs to be taught
    to the robot for it to accurately weld the
    desired path
  • the torch travel speed along the seam, as well as
    the torch stand-off and orientation with respect
    to the seam, can all be controlled in real time.
  • this enables highly sophisticated weld process
    control techniques to be implemented, including
    feed-forward (adaptive) fill, and real-time
    closed-loop feedback control of the welding
  • fixturing and setup are thus greatly simplified,
    which facilitates low volume industrial
  • indeed, since path programming is also
    eliminated, this system can effectively be used
    for one-off production

  • Other successful examples of machine vision in
    traditional robotics
  • pick and place
  • parts location
  • parts recognition
  • parts inspection

  • Some non-robotic applications of machine vision

skin lesion classification
hand gesture recognition
defect detection in welds
facial recognition
hand writing recognition
  • More applications of machine vision

traffic surveillance and speed checks
lane and neighbouring vehicle detection
  • Advanced topics in machine vision
  • colour vision
  • three dimensional vision
  • analysis of dynamic scenes
  • analysis of complex scenes
  • robust analysis of any scene !
  • robust analysis of colour, 3D, dynamic complex
    scenes !

  • Implementation issues why isnt the use of
    machine vision more widespread ?
  • A number of factors have delayed the practical
    development of intelligent sensors based
  • These can be most easily divided into the
    inadequacies of today's robots and the limited
    performance of current vision systems.
  • These problems are compounded by the analytical
    and computational complexity of both manipulator
    control and sensory data interpretation.
  • Robot control systems are difficult to analyse
    and design and tend to employ very simple robot
    models so that trajectories can be computed
    rapidly enough so as not to degrade the robot arm
    dynamic performance.
  • In addition robot geometry may be slightly
    different from the model, and therefore the
    actual end-effector position may differ from the
    desired position.

  • Implementation issues why isnt the use of
    machine vision more widespread ?
  • The performance and effectiveness of current
    vision systems remain too limited, in many cases,
    for real-time sensory feedback applications.
  • The major requirements for a real-time sensory
    feedback system is the development of three
    dimensional sensing and the ability to carry out
    dynamic scene analysis.
  • The current limitations on such a system are the
    data processing rates required, the data volume
    per image and the extremely complex data
  • These limitations drastically affect real-time
    dynamic manipulator control and, until very
    recently, have been orders of magnitude slower
    than manipulator dynamics.
  • The primary technical difficulties to overcome
    are the development of high speed integrated
    circuits, image processing architectures and
    optimised image data flow.

Week 11 External sensors and intelligent
robots1) Sensors for robotics 2) Machine
vision overview of processes and
components 3) Image processing image
enhancement image analysis image
understanding4) Implementation
issues5) Applications6) Advanced applications
Tutorial questions
9.1 Why is sensory development important to
robotic design ? 9.2 Consider a simple task and
discuss how the human intelligently
co- ordinates perception and action. Contrast
this with the manner in which a robot would
carry out the task. 9.3 Discuss the advantages
that vision systems have over conventional
sensor systems. 9.4 Describe the main functional
elements in a machine vision system with the aid
of diagrams. Go on to describe the major
processes in an image processing system. 9.5 In
a robot-based production engineering scenario
what are the main limitations that have
prevented the full implementation of vision
based robotics? 9.6 Describe briefly sensors for
use with robots and, in particular, the
development of machine vision. 9.7 Write a
short essay on the use of machine vision in robot
applications using sketches where appropriate.
Write a Comment
User Comments (0)