Towards%20Autonomous%20Data%20Ferry%20Route%20Design%20in%20Delay%20Tolerant%20Networks

1
Towards AutonomousData Ferry Route Design
inDelay Tolerant Networks

• Daniel Henkel, Timothy X Brown
• University of Colorado _at_ Boulder
• WoWMoM/AOC 08
• June 23, 2008

TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAA
2
Familiar Dial-A-Ride
Dial-A-Ride curb-to-curb, shared ride
transportation service
request 1
request 2
hospital
request 3
The Bus
request 4
school

• Receives calls
• Picks up and drops off passengers
• Transport people quickly !

request 5
depot
Optimal route not trivial !
3
In context Dial-A-UAV
Complication infinite data at sensors
potentially two-way traffic
Delay tolerant traffic!
Sensor-1
Sensor-3
Sensor-5
Monitoring Station
Sensor-2
Sensor-6
Sensor-4

• Sparsely distributed sensors, limited radios

• TSP solution not optimal
• Our approach Queueing and MDP theory

4
TSPs Problem
Traveling Salesman Solution
A
B
UAV
hub

• One cycle visits every node
• Problem far-away nodes with little data to send
• Visit them less often

dA
dB
fA
fB
5
Queueing Approach
Goal Minimize average delay

• Idea express delay in terms of pi, then minimize
  over set pi
• pi as probability distribution
• Expected service time of any packet
• Inter-service time exponential distribution with
  mean T/pi
• Weighted delay

A
B
UAV
fB
fA
pA
pB
dB
dA
pC
C
hub
pD
dC
dD
fC
D
fD
6
Solution and Algorithm

• Probability of choosing node i for next visit

Implementation deterministic algorithm 1. Set
ci 0 2. ci ci pi while maxci lt 1 3. k
argmax ci 4. Visit node k ck ck-1 5.
Go to 2.
Pretty simplistic view of the world ! Random
selection ignores many parameters.
7
Theres More to It!

• New perspective
• States
• people waiting at location
• Varying of calls (daytime)
• Current bus location
• Actions
• Drive to a location
• Goal
• Short passenger wait time

request 1
request 2
request 3
request 4
request 5
depot
? Generally unknown environment
8
Promising Technique

• Reinforcement Learning (AI technique)
• Learning what to do without prior training
• Given high-level goal NOT how to reach it
• Improving actions on the go
• Features
• Interaction with environment
• Concept of Rewards Punishments
• Trial Error Search
• Example riding a bike

9
The Framework

• Agent
• Performs Actions
• Environment
• Gives Rewards
• Puts Agent in situations called States
• Goal
• Learn what to do in a given state (Policy)

The Beauty Learns model of environment and
retains it.
10
Markov Decision Process
Series of States/Actions

• Markov Property
• reward and next state depend only on the
  current state and action, and not on the history
  of states or actions.

11
MDP Terms

• Policy Mapping from set of States to set of
  Actions
• Sum of Rewards (return) from this time onwards
  
• Value function (of a state) Expected return when
  starting in s and following policy p. For an MDP
• Solution methods
• Dynamic Programming, Monte Carlo simulation
• Temporal Difference learning

12
UAV Path Planning
A
B
?A
?B
H
F
D
C
?D
?C

• State tuple of accumulated node traffic, here
• Actions round trip through subset of nodes,
  e.g., A, B, C, D, AB, AC,DCBA

13
Reward Criterion

• Reward

14
Learning

• Temporal Difference Learning
• Recursive state value approximation
• Convergence to true value as

15
Paths
16
Simulation Results

• RR Round Robin (naive)
• STO Stochastic Modeling

• TSP Traveling Salesman solution
• RL Reinforcement Learning

17
Conclusion/Extensions

• Shown two algorithms to route UAVs
• RL viable approach

Extensions

• Structured state space
• Action space (options theory)
• Hierarchical structure / peer-to-peer flows
• Interrupt current action and start over
• Adapt and optimize learning method

18

• Soccer - Euro Cup 2008

Wednesday, 1145am (PST) Germany Turkey 4 1

19
Research Engineering Center for Unmanned
Vehicles (RECUV)
Questions
Research and Engineering Center for Unmanned
Vehicles University of Colorado at
Boulder http//recuv.colorado.edu
The Research and Engineering Center for Unmanned
Vehicles at the University of Colorado at Boulder
is a university, government, and industry
partnership dedicated to advancing knowledge and
capabilities in using unmanned vehicles for
scientific experiments, collecting geospatial
data, mitigation of natural and man-made
disasters, and defense against terrorist and
hostile military activities.