Showing posts with label robotics. Show all posts
Showing posts with label robotics. Show all posts

Saturday, April 13, 2019

Interoperable autonomous mining

I was speaking to someone recently who asked me why I stopped blogging - and timely enough, the Future of Mining conference happened to record a recent short speech and published it on youtube. The video describing Epiroc Autonomous Pit Viper blasthole drills performance and the future of autonomous interoperability is below. I'm not used to summarising so much in 10 minutes, but I briefly discuss the business impact of autonomous blasthole drills, increased production, increased quality, and more consistent performance. I make the argument that autonomy is a critical first step prior to realising an integrated operation: essentially you can not achieve consistent real-time control of your fleet if your reliant on the variability of manned operators, and without the information (rich data plus semantics) produced by autonomous equipment. To achieve the objective of a zero-entry (no people) integrated mine you will need to automate all the mining equipment, and waiting for a single supplier to achieve this means delayed realisation of the business case. Collaboration of multiple suppliers enables best-of-breed and flexibility ensuring we see a fully autonomous and integrated mine sooner.

I'll start blogging again now - covering some of my backlog from the last few years on autonomous mining, but also back to some technical topics as well. Stay tuned.

Sunday, September 30, 2012

Mining Robotics: An overview survey

Robotics is typically associated with manufacturing robotics (e.g. PUMA arm), military robotics (e.g. Predator UAV), and more recently consumer robots (e.g. Roomba), medical/healthcare (e.g. Da Vinci) and the automotive industry (e.g. driverless cars). Not many are aware of the prevalence of robotics in the mining industry and the steps the industry has taken towards automation and autonomous robots.
Autonomous mining

The mining industry is a world-leader in autonomy, for example Rio Tinto's Western Australia operations has the worlds largest fleet of autonomous vehicles (150 autonomous trucks) - significantly larger than any operational system in the military. Rio's Western Australia operations are all controlled from a operation centre, which controls 40 mines, 30 pits, trains, power stations, and ports all based thousands of kilometres away. In terms of data, the WA system generates around 2.4 terabytes per minute of data. There is quite a lot of intelligence and innovation involved.

Overall, mining can be broadly broken up into a few key phases:
Mining process
  • Exploration, assessment and planning. In this phase, new resources are identified and a new mine site is designed and constructed.
  • Drill and blast, material is extracted from the ore.
  • Load and haul, material is taken from the point of extraction to the processing plant.
  • Processing, where the material is converted/crushed into a more useful (sellable) form.
  • Transportation, where the product is loaded and transported, usually via rail to a port and then on to a ship to its final destination.
  • Stockpiling, occurs at various points in varying quantities where appropriate.
At each of these steps some kind of machine is involved, and I'll give you a brief overview of the machines and some of the relevant research or commercial automation systems available.

Exploration and remote sensing is a massive research area in itself in other industries, and mining is no different. UAV's are seeing use in aerial surveys on mine sites, with large data sets fusing visual (photogramatery), infrared, LIDAR, InSAR, gradiometry, seismic and other geodesic measurements.
West Angelas mine LIDAR scan
On the ground new sensor fusion systems are being developed to classify the mine and ore structure and to identify the richest ore deposits. Combining all this data into an overall mine model is a difficult machine learning task. The Rio Tinto Center for Mine Automation are doing active research in this field, and the Gatewing X100 is an example of an UAV used for mapping in mining.


Drill and blasting is a mining-specific operation and there has been significant advancement in robotics in this area due to the operational hazards involved with this line of work. Robots can accurately drill holes that won't collapse and are easy to load, and Atlas CopCo and Flanders both have commercial automation systems for drilling which are well on their way to delivering autonomous drill rigs in the near future (trial drilling systems have been in use on production sites since 2008). Atlas Copco first started their work in automated drill rigs in the 1980's and now has over 2,500 machines running their control system technology.

The load and haul stage is perhaps the most interesting as it is the first area where autonomous vehicles are used in regular production environments. Whilst autonomous loading is still an area of research (See these CSIRO projects on dragline and shovel loading automation), there are plenty of commercial automation systems for haul vehicles.
This includes CAT Minestar Command, Atlas-Copco Scooptram, Sandvik Automine, Komatsu Frontrunner. Mineware provide shovel and dragline automation systems, with LIDAR systems that build digital terrain maps on the fly. Autonomous Solutions has a number of autonomous vehicles, including trucks and dozers.
Continuous miners and long wall mining have seen multiple automation systems including commercial systems from Eickhoff and CAT. Excavators are no stranger to automation, CMU automated excavators and truck dumping back in the late 90's, and work is ongoing at PWRI in Japan and Hyundai research. The range of commercially available autonomous mining vehicles put military UGV's and automotive companies to shame.


Transmin Rocklogic
Processing plants have been fully automated, although for many metals, (e.g. iron ore), there isn't much too the process in the first place. Companies such as Metso have fully automated crushers, conveyors, and also include computer vision systems to identify and classify rocks/froth/bubbles, etc. FLSmidth and Calibre Transmin have developed automation systems for rock breakers are available allowing the rockbreaker to automatically park and deploy. In-Pit Crushing and Conveying (IPCC) systems allow parts of the plant to be mobile, and even these systems have been largely automated by companies such as Sandvik.

Autonomous train
Transporting material from the mine is usually performed by a train, and autonomous trains have been around for a while. In fact, LKAB have been running driverless trains since the 1970's. The main difference in modern mining applications being that the goal is now fully autonomous operation, and that the trains can stretch many kilometres in length, making control a more difficult problem. Major miners such as Rio Tinto are automating the trains in Western Australia, with companies such as Ansaldo STS and New York Air Brakes providing the technology.

Finally, with stockpiling Stacker-Reclaimers have been automated, with companies such as ThyssenKrupp and iSAM leading the way


Rio Tinto - Remote Operation Center
Overall there are a large amount of automated and autonomous mining equipment available, and projects such as Rio Tinto's mine of the future at West Angelas and Yandicoogina sites, Vale's Carajas Serra Sul S11D site and Nautilus's Solwara underwater mining are all pushing towards fully autonomous sites where we may see no humans involved in operating future mine sites.


So if you want to find out more about robotics and automation research in mining there are a few great places to start:
The future of mining is autonomous robots, and we are well on our way!

Thursday, June 07, 2012

GPU Technology Conference 2012

nVidia's GPU Technology Conference is over, and a number of presentation slides have been uploaded. There were a quite a few interesting talks relating to graphics, robotics and simulation:
  • Simon Green from nVidia and Christopher Horvath from Pixar presented 'Flame On: Real-Time Fire Simulation for Video Games'. It starts with a recent history of research on CG fluid systems, and gives five tips on better looking fire: 1. Get the colors right (e.g. radiation model), 2. Use high quality advection (not just bilinear filtering), 3. Post process with glow and motion blur. 4. Add noise. 5. Add light scattering and embers. They then go into more detail on Tip #1 looking at the physics behind the black-body radiation in a fire, and the color spectrum.
  • Elmar Westphal of PGI/JCNS-TA Scientific IT-Systems presented 'Multiparticle Collision Dynamics on one or more GPUs', about multiparticle collision dynamics GPU code. He starts by explaining the overall algorithm, and explaining step-by-step what performs well on the GPU. Specific GPU optimisations explained include spatial subdivision lists, reordering particles in memory, hash collisions, and finally dividing workload between multiple GPU's. An interesting read.
  • Michal Januszewski from the University of Silesia in Katowice introduces 'Sailfish: Lattice Boltzmann Fluid Simulations with GPUs and Python'. He explains lattice boltzmann fluid simulation, and some of the different configurations of lattice connectivity and collision operators. Moves into code generation examples, and gives a brief explanation of how the GPU implementation works.
  • Nikos Sismanis, Nikos Pitsianis and Xiaobai Sun (Aristotle University, Duke University) cover 'Efficient k-NN Search Algorithms on GPUs'. Starts with an overview of sorting and K-Nearest Neighbour (KNN) search algorithm solutions, including ANN (approximate NN) and lshkit and moves into results including a comparison of thrust::sort with Truncated Bitonic sort. Software is available at http://autogpu.ee.auth.gr/.
  • Thomas True of nVidia explains 'Best Practices in GPU-Based Video Processing' and covers overlapping copy-to-host and copy-to-device operations, and an example of processing bayer pattern images.
  • Scott Rostrup, Shweta Srivastava, and Kishore Singhal from Synopsys Inc. explain 'Tree Accumulations on GPU' using parallel scatter, parallel reduce and parallel scan algorithms.
  • Wil Braithwaite from nVidia presents an interesting talk on 'Interacting with Huge Particle Simulations in Maya using the GPU'. He begins with a brief runthrough of the workings of the CUDA SPH example, and then moves onto the particle system including Maya's body forces (uniform, radial, vortex), shape representations (implicit, covex hull, signed distance fields, displacement maps), collision response, SPH equations, and finally data transfer. Ends with a brief overview of rendering the particles in screen space. Neat.
  • David McAllister and James Bigler (nVidia) cover the OptiX internals in 'OptiX Out-of-Core and CPU Rendering' including PTX code generation and optimisation, and converting the OptiX backend to support CPU's via Ocelot and LLVM. An interesting result, LLVM does better at optimising "megafunctions" than small functions, but not entirely unexpected given how LLVM works. The presentation finishes with an overview of paging and a tip on bounding volume heirarchies. Good to see Ocelot in the mainstream.
  • Eric Enderton and Morgan McGuire from nVidia explain 'Stochastic Rasterization' (ala 'screen door transparency' rendering) via MSAA for motion blur, depth of field and order-independent transparency, by using a geometry shader to bound the shape and motion of each tri in screen space, and setting up the MSAA masks. Nice.
  • Cliff Woolley presents 'Profiling and Tuning OpenACC Code' (by adding pragmas to C / Fortran code, ala OpenMP) using an example of Jacobi iteration, and there were a number of other talks on the topic.
  • Christopher Bergström introduced 'PathScale ENZO' the alternative to CUDA and OpenCL.
  • Phillip Miller from nVidia did an broad coverage of 'GPU Ray Tracing'. He starts with a myths and claimed facts on GPU raytracing, highlights some commercial GPU raytracers (and the open source OpenCL LuxRenderer) and goes into some details that are better explained in the OptiX Out-of-Core presentation.
  • Phillip Miller follows with 'Advanced Rendering Solutions' where he takes a look at nVidia's iray, and where they believe they can introduce new capabilities for design studios and find a middle ground with re-lighting and physcially based rendering.
  • Peter Messmer presents 'CUDA Libraries and Ecosystem Overview', where he provides an overview of the linear algebra cuBLAS and cuSPARSE libraries performance, then moves to signal processing with cuFFT and NPP/VSIP for image processing, next is random numbers via cuRAND and finally ties things up with Thrust.
  • Jeremie Papon and Alexey Abramov discuss the 'Oculus real-time modular cognitive visual system' including GPU accelerated stereo disparity matching, likelihood maps and image segmentation with a parallel metropolis algorithm.
  • Jérôme Graindorge and Julien Houssay from Alyotech present 'Real Time GPU-Based Maritime Scenes Simulation' beginning with ocean simulation and rendering from FFT based wave simulation using HF and LF heightmap components. They then cover rendering the mesh, scene illumination and tone mapping, and a sneak peak at boat interaction. The ocean simulation video is neat.
  • Dan Negrut from the Simulation-Based Engineering Lab at the University of Wisconsin–Madison gives an overview of the labs multibody dynamics work in 'From Sand Dynamics to Tank Dynamics' including friction, compliant bodies, multi-physics (fluid/solid interactions), SPH, GPU solution to the cone complementary problem, ellipsoid-ellipsoid CCD, multi-CPU simulation, and finally vehicle track simulation in sand. Wow. Code is available on the Simulation-Based Engineering Lab website.
  • Max Rietmann of USI Lugano looks at seismology (earthquake simulation) in 'Faster Finite Elements for Wave Propagation Codes' and describes parallising FEM methods for GPUs in SPECFEM3D.
  • Dustin Franklin from GE introduces GE's MilSpec ruggedised Kepler-based GPU solutions and Concurrent Redhawk6 in 'Sensor Processing with Rugged Kepler GPUs'. Looks at some example applications including hyperspectral imaging, mosaicing, 360 degree vision, synthetic aperture radar processing, and space-time adaptive processing for moving target identification.
  • Graham Sanborn of FunctionBay presents 'Particle Dynamics with MBD and FEA Using CUDA' and gives a brief overview of their combined CPU/GPU multi-body FEA system and briefly describes the contact, contact force, and integration steps.
  • Ritesh Patel and Jason Mak of University of California-Davis cover the Burrows-Wheeler Transform, Move-to-Front Transform and Huffman Coding in 'Lossless Data Compression on GPUs'. They find merge sort for BWT performs best on the GPU, explain the parallel MTF transform and Huffman in illustrative detail and tie things up with benchmarks, unfortunately GPU is 2.78x slower than CPU.
  • Nikolai Sakharnykh and Nikolay Markovskiy from NVIDIA provide an indepth explanation of their GPU implementation of solving ADI with tridiagonal systems in '3D ADI Method for Fluid Simulation on Multiple GPUs'.
  • Enrico Mastrostefano, Massimo Bernaschi, and Massimiliano Fatica investigate breadth first search in 'Large Graph on multi-GPUs' and describe how best to parallelise it across multiple GPU's by using adjacency lists and level frontiers to minimise the data exchange.
  • Bob Zigon from Beckman Coulter presents '1024 bit Parallel Rational Arithmetic Operators for the GPU' and covers exact 1024 bit rational arithmetic (add,sub,mul,div) for the GPU. Get the 1024 bit arithmetic code here.
  • Roman Sokolov and Andrei Tchouprakov of D4D Technologies discuss 'Warped parallel nearest neighbor searches using kd-trees' where they take a SIMD style approach by grouping tree searches via voting (ballot)
  • David Luebke from nVidia takes a broad look at CG in 'Computational Graphics: An Overview of Graphics Research @ NVIDIA' and provides an overview of research which is featured in a number of previous talks and other GTC talks including edge aware shading, ambient occlusion via volumes and raycasting, stochastic rendering, improved image sampling and reconstruction, global illumination, and CUDA based rasterization.
  • Johanna Beyer and Markus Hadwiger from King Abdullah University of Science and Technology discuss 'Terascale Volume Visualization in Neuroscience' where each cubic mm of the brain scanned with an electron microscope generates 800 tereabytes of data. The idea here is to leverage the virtual memory manager to do all the intelligent caching work, rather than a specialised spatial datastructure for the volume rendering.
  • Mark Kilgard introduces the NV_path_rendering extension in 'GPU-Accelerated Path Rendering', and demonstrates using the GPU to render PDF, flash, clipart, etc. Contains some sample code.
  • Janusz Będkowski from the Warsaw University of Technology presented 'Parallel Computing In Mobile Robotics For RISE' a full GPGPU solution for processing mobile robot laser scan data through to navigation. Starts with data registration into a decomposed grid which is then used for scan matching with point-to-point Iterative Closest Point. Next is estimating surface normals using principle component analysis, demonstrated on velodyne datasets. This is used to achieve point-to-plane ICP and he demonstrates a 6D SLAM loop-closure. Finishes it all off with a simple gradient based GPU path planner.
Note that in recent days more presentation PDF's have been uploaded so there is still plenty to look through, and with all the content it's difficult to look through it all - take a look yourself! I'll leave you with a video from the GTC 2012 keynote on rendering colliding galaxies:

Tuesday, May 08, 2012

Dynamic Window Algorithm motion planning

Most robots have a set of navigation algorithms for motion planning that execute at different frequencies, global path planners (e.g. A*, ~0.1 Hz), mid-level path deformation (e.g. elastic band, ~5Hz), and collision / obstacle avoidance algorithms (~20Hz), which will be the last step before actuator control.
For MAGIC 2010, we used the Dynamic Window Approach.
(Note the ROS navigation stack offers the same algorithm configurations, but of-course didn't exist at the time we had to develop the WAMbot codebase). There are three common approaches used for local trajectory planning:
  • Potential-field based, where each obstacle has an obstacle 'force field' for repelling the robot, and the goal has a attraction field. (A similar approach is 'Vector-fields', and Virtual Force Field)
  • Dynamics based, where the algorithm consider the robots dynamics in calculating a solution. (e.g. Velocity Obstacles and Dynamic Window Approach)
  • Sampling based, where various collision free states are sampled and then combined. (e.g. Reachability graph, Probabilistic roadmaps)
The Dynamic Window Approach is a velocity-based local planner that calculates the optimal collision-free ('admissible') velocity for a robot required to reach its goal. It translates a cartesian goal (x,y) into a velocity (v,w) command for a mobile robot.

There are two main goals, calculate a valid velocity search space, and select the optimal velocity. The search space is constructed from the set of velocities which produce a safe trajectory (i.e. allow the robot to stop before colliding), given the set of velocities the robot can achieve in the next time slice given its dynamics ('dynamic window'). The optimal velocity is selected to maximize the robots clearance, maximize the velocity and obtain the heading closest to the goal.

Its easier to explain if we look at the code first. In pseudo-code the DWA is:
 
BEGIN DWA(robotPose,robotGoal,robotModel)
   desiredV = calculateV(robotPose,robotGoal)
   laserscan = readScanner()
   allowable_v = generateWindow(robotV, robotModel)
   allowable_w  = generateWindow(robotW, robotModel)
   for each v in allowable_v
      for each w in allowable_w
      dist = find_dist(v,w,laserscan,robotModel)
      breakDist = calculateBreakingDistance(v)
      if (dist > breakDist)  //can stop in time
         heading = hDiff(robotPose,goalPose, v,w)
         clearance = (dist-breakDist)/(dmax - breakDist)
         cost = costFunction(heading,clearance, abs(desired_v - v))
         if (cost > optimal)
            best_v = v
            best_w = w
            optimal = cost
    set robot trajectory to best_v, best_w
END

Now to explain:
  1. First, we can calculate the desired velocity to the goal based on our current position, and the destination. (e.g. go fast if we are far away, slow if we are close. Use Equations of Motion, see Circular motion for a mobile robot).
  2. Select the allowable velocities (linear 'v', and angular 'w') given the vehicles dynamics, e.g. allowable_v ranges from the current velocity subtract the robots maximum deceleration * timeslice to the current velocity plus the robots maximum acceleration * timeslice, or more compact: [v-a.t,v+a.t], likewise for angular velocity.
  3. Search through all the allowable velocities
  4. For each velocity, determine the closest obstacle for the proposed robot velocity (i.e. collision detection along the trajectory)
  5. Determine if the distance to the closest obstacle is within the robots breaking distance. If the robot will not be able to stop in time, disregard this proposed robot velocity.
  6. Otherwise, the velocity is 'admissible', so we can now calculate the values required for the objective function. In our case, the robots heading and clearance.
  7. Calculate the 'cost' for the proposed velocity. If the cost is better than anything else so far, set this as our best option.
  8. Finally, set the robots desired trajectory to the best proposed velocity.
If you've read all the other posts, you should now know enough to implement your own mobile robot navigation system, or at least understand a bit more about the algorithms in the ROS navigation stack.

Thursday, March 29, 2012

Elastic Band - Realtime pathfinding deformation

I've covered global path planners before including A* path planning and Dijkstra. These path planners will give you a result, but if something alters the desired path (such as an object crosses the path, which you need to avoid) you need to recompute the whole path again. Algorithms such as D* lite allow efficient re-computation. However, the elastic band method allows the existing path to be used, and just adjusted to handle deformations. Or, more formally, elastic band path planning enables realtime modifications to a precomputed path that consider additional obstacles (or cost functions) not considered during the original paths computation.
Elastic band - Global path in blue, bubbles as colored circles,
 and the red path is the one generated by the elastic band algorithm.

The easiest way to describe it in games terminology is to have an entire path turned into particles that follow boid-like (flocking) rules. The best part, is that since it is essentially based on particles, means that you can re-use code from a particle system or physics engine you may have available. If your from the games background, you can use player/entity interchangeably with my use of the word 'robot'.

The 'elastic band' is a created from a set of 'bubbles'. A bubble contains a radius, a position, the coordinates of the closest obstacle to the bubble and a velocity.

The way it works is that an 'elastic band' is initialised with the partial path from the global path planner. The centre of each bubble is assigned to a point along the global path. (For optimisation purposes, a sparse version of the global path is typically used).

If the elastic band does not contain the robots current position, then the robots present position is inserted, provided it will align with the desired path.

For every bubble in the path we determine the total external and internal forces acting on it, and apply them and create/remove bubbles as necessary.
This is done by:
  • Creating a new bubble if the distance to the next bubble is greater than a predefined constant. In effect this constant determines the discretization of the path.
  • Determining the closest obstacle to the bubble and calculating the radius. This is achieved by using a spatial partitioning algorithm to determine the closest obstacle by its Euclidean distance. The radius is assigned to this distance, limited to a maximum.
  • Calculating the external repulsion force from the closest obstacle. The magnitude of this force is scaled to be a function of its distance limited by an upper bound, such that closer obstacles exert a greater pressure. (i.e. (max - r) / r )
  • Calculating the internal cohesion force from the previous and following bubbles in the path. This is simply the sum of the normalised vectors to the previous and following bubbles centres.
  • Updating the bubble state by applying the forces and velocities to calculate the new bubble position. I used a higher order integrator with hand-tuned dampening terms. In addition, a low pass filter is applied to the position update to reduce the influence of the forces.
Finally, bubbles are removed if they are too close to each other. The entire band is then checked for continuity, ensuring that each bubble is greater than the robots size. If this criteria is not fulfilled the robot is halted, as it will not be able to get to its destination without colliding.

The bubble that is a set distance infront of the robot is selected to generate the next trajectory input. The angle between the current bubble and the next bubble is calculated and used to modulate the vehicles desired velocity. This causes the vehicle to reduce its speed when it approaches a sharp turn.

In pseudo-code the algorithm is as follows:
eb[0] = path.start
i = 0
for each point along path
  if dist(eb[i],point)> c
    eb.add(point)

while robot not at path.goal
  for each bubble b in eb
    f_e = b.pos – closestObstacle(b)
    f_i = (b.pos – eb[b.i – 1].pos) + (b.pos – eb[b.i + 1].pos)
    b.integrate(f_e+f_i,b.vel) //update velocity from forces
    b.dampen(b.vel) //filter state
    b.integrate(b.vel,b.pos)
    dist = abs(b.pos – eb[b.i + 1].pos)
    if dist > c
      eb.add(midpoint(b,eb[b.i+1]))
    if dist < c_min
       eb.remove(b)
  tmp_goal = eb.closestbubble(path.robot + lookahead) //set goal from robot position
  set robot trajectory to tmp_goal
Thanks to Sushil who did all the hard work in implementing this for MAGIC2010.

Tuesday, March 20, 2012

Collision Avoidance in Mining Conference

I gave a presentation on the Transmin Rocklogic rockbreaker collision avoidance system at the 3rd Collision Avoidance in Mining Conference held in Fremantle.

The speakers were:
  • Simon Ridge - Department of Mines and Petroleum: provided an overview of regulation and risk management in WA.
  • Stewart Bell - Department of Employment, Economic Development and Innovation: provided an overview of collision incidents in Queensland, and made an announcement indicating Queensland may make collision avoidance technology mandatory.
  • David Quayle - Industrea: discussed systems integration issues in mining, and contrasted it with aerospace and the automotive industry.
  • Patrick Glynn - CSIRO: provided an overview of the development of collision avoidance technology in mining, and discussed CSIRO's old LHD automation project.
  • Luke Schelosky - SafeMine: discussed SafeMine's proximity / collision detection products.
  • Marcus Punch: discussed functional safety standards (e.g. SIL), the standards that the industry is moving towards, and the "Proven-In-Use" clause for the latest 61508.2 standard.
  • Matthew Watson - Rio Tinto Iron Ore: discussed issues in rolling out technology and how it affects operators ('users'), managers, and maintenance personal.
  • Rhys Sherborne - APS: gave an overview of Automated Positioning System machine guidance products.
  • Harland Attwood - Javpac: introduced a new physical barrier product designed to separate heavy vehicle traffic.
  • Peter Woodford - LSM: discussed cameras and operator visibility.
  • Elliot Duff - CSIRO: gave an overview of projects at the Autonomous Systems Laboratory.
  • Nicky Guenther - SICK: presented a new range of SICK products for proximity detection and provided a comparison with RADAR products.
  • Adrian Boeing - Transmin: Rocklogic collision avoidance system, and installation case study.
  • Robin Burgess-Limerick - UQ: Operator cognitive load.
  • Natalie Tindale - MB solutions: Operator fatigue.
For me the highlight was CSIRO's Autonomous Systems Labs robotics and SLAM work, particularly on the North Parkes mine site. Since they don't have a YouTube video of their work on the ASL channel, here is something similar:

Thursday, January 12, 2012

Kinect viewer with HTML5 canvas

I wanted to create a web-based Kinect viewer that would work on multiple platforms, including Android mobile phones and iPhones. To this end, I created a HTML5 canvas particle viewer for Kinect data using my very limited javascript knowledge and a lot of Mr. doob's knowledge.
Kinect HTML5 particle image

Take a look at the HTML5 Kinect particle image viewer.

Using libfreenect I captured the depth and color data, used Imagemagik and an Automator script to process and stich the images, and threejs to render.

You can extract data from an image by rendering it to a hidden canvas element, and then reading the canvas image data. To avoid errors, the image data should only be read after the full image has been loaded, as per the callback function in loadTexture.

function getImageData( image ) {
var canvas = document.createElement( 'canvas' );
canvas.width = image.width;
canvas.height = image.height;
var context = canvas.getContext( '2d' );
context.drawImage( image, 0, 0 );
return context.getImageData( 0, 0, image.width, image.height );
}

function getPixel( imagedata, x, y ) {
var position = ( x + imagedata.width * y ) * 4, data = imagedata.data;
return { r: data[ position ], g: data[ position + 1 ], b: data[ position + 2 ], a: data[ position + 3 ] };
}

var texture, imagedata;
texture = THREE.ImageUtils.loadTexture( "textures/sprites/spark1.png", new THREE.UVMapping(), function ( event ) {
imagedata = getImageData( texture.image );
} );


To run the system locally, avoiding some of Chromes very annoying security settings, you can just use python to run a simple webserver:

python -m SimpleHTTPServer


I recorded a short data sequence and rendered to sprites to create this final 3d home move with the Kinect (too slow for my phone to render). I believe this is the first web-based kinect video ever made, youtube3d anyone?



Animated 3D kinect data with HTML sprites

Next time, I might try WebGL...

Tuesday, November 29, 2011

Remotely Operated Mining

I gave a presentation on the Transmin Rocklogic rock breaker automation system at the Remotely Operated Mining Projects conference. All in all it was a fairly decent conference focused around the theme of remote operations for mining, focusing predominantly on integrated planning, remote operation technology, and the design and construction of remote operation centers.

Talks were given by:
  • Greg Lilleyman, Rio Tinto - an overview of remote operations
  • Matthew Michalewicz, SolveIT - integrated planning for mature organizations, interesting overview of some of the wine industry supply chain issues
  • Patrick Warrand, Wesfarmers Premier Coal - a open overview of issues for 'small' mining operations (Note: Small is of course a relative term compared to the massive operations of BHP and Rio)
  • Jeremy Newman, Schneider Electric - a suppliers point of view to automation
  • Jock Cunningham, CSIRO - an overview of CSIRO activity in mining, and their SLAM technology. (WAMBOT was multi-agent SLAM, CSIRO does automotive SLAM and a novel handheld indoor SLAM 'Zebedee')
  • Brendon, Leica Geosystems - an overview of Autonomous Solutions Inc (they do lots of autonomous systems for military and agriculture), and Locata - a GPS alternative
  • David Beck, Beck Engineering - a case study of Simulation Aided Engineering
  • Adrian Boeing, Transmin - remotely operated rockbreaking
  • David Cavanagh, Integrated Energy - an overview of remote operation center designs for oil and gas, and mining
  • Lucas van Latum, Modular Mining - Goal orientated planning and the Observe-Orientate-Decide-Act loop
  • Reuben Kooperman and Peter Hunt - architecture for Remote Operation Centers (or Integrated Planning Centers) - including two buildings I go past on a regular basis and never fully realised what goes on inside - the City of Stirling building, and the ECU Joondalup library.
  • Mike Lomman, OpX Solutions - planning considering bottlenecks
  • Andrew Jessett, MineWare - realtime dragline and shovel monitoring and support/maintenance orientated ROC's.
  • David Hattrick, Oracle Corporation - Oracles approach to managing mining data and information
  • Richard Taylor, Concept AV - Video walls
I put the Rocklogic animation up on youtube, it's pretty neat. Gives a short overview of the systems features - Remote operation, collision avoidance, automated movements (auto-park, auto-deploy), etc.

Sunday, September 25, 2011

Robotics competitions and courses

Sounds like it is competition season again.
  • UAV outback challenge where an unmanned ariel vehicle needs to find a person and drop supplies off to them. Registrations are due on the 27th September!
  • NASA Sample Return Centennial Challenge, a 1.5 million dollar prize competition to have robots autonomously explore a region, collect samples, and bring them back to base. Registration due in January, with the competition set for June. Only 8 months before the deadline when they announced the competition details - what are they thinking?
  • The ICRA solutions in perception challenge (May 2012), this doesn't seem too difficult - Identify and classify a set of objects, where you are given some training data/time. Use any sensor(s) you like.
  • Australian CiSRA ExtremeImaging competition an interesting computer-photography competition, due October this year, finals in November.
Many more are listed at the robot contests and competitions FAQ. To get you up to speed with robotics, there are some great free courses being run. The stanford Machine Learning course and the Introduction to AI class are both free to register, and start now! The ESA summer of code in space already started last month, but it is worth keeping an eye on for next time.

Tuesday, September 06, 2011

AIMEX

I visited the Asia-Pacific International Mining Exhibition (AIMEX), held every four years. There was a lot of interesting technology on display, with a strong presence of simulators, drills and long-wall gear. Being mining, a lot of very large machines were on display, and the exhibition hall was huge. Automation also had a small presence.

Atlas-Copco had a set of their autonomous drill rig simulators on display, and I managed to get a chance to go through the process. Drilling is a non-trivial task, and it takes at least two weeks for an operator to familiarize themselves with the process. The autonomous drill system certainly makes it easier to operate the equipment, but the drill holes still have to be lined up manually - arguably the most challenging task.
Atlas Copco Autonomous Drill console

Eickhoff had a demonstration of their automated long-wall system, that uses the Indurad radar system.

CAT were displaying some of their new autonomous equipment, soon to be installed in
Western Australia at FMG's Solomon mine. CAT is always a slow mover to adopt technology but it is interesting to see how fast this technology is being adopted by the major Australian miners, now including Rio Tinto, BHP, Newcrest and FMG.
Eickhoff longwall automation 

Below is a video of the CAT system I captured at AIMEX. Apparently this system actually does realtime path planning and adjustment as opposed to following a pre-programmed path like other (Sandvik) systems.


You can watch some more videos of autonomous mine vehicles on youtube:

Tuesday, August 16, 2011

Kinect and OpenKinect on Apple Mac OSX

Adrian, in Kinect-3D!
The Kinect is a revolutionary sensor for robotics - the low price point has seen an explosion of interest and applications in this field. (Hopefully we will see the same for the Neato XV11 low cost laser scanner one day). I was very impressed by the data the Kinect returned, it is a fantastic device.

To get started, just follow the OSX install instructions at OpenKinect. You will need MacPorts. A copy of the instructions is here:
sudo port install git-core
sudo port install libtool
sudo port install libusb-devel
git clone https://github.com/OpenKinect/libfreenect.git
cd libfreenect
mkdir build
cd build
ccmake ..
cmake ..
make
sudo make install
(Press 'g' to generate your cmake make files). Very simple.

Now check under system profiler to see that the Kinect device is detected.

I had the following error when I tried to read from the device:
Isochronous transfer error: 1 

Thankfully, this was easy to solve with a fully self-contained installer from AS3 kinect OSX download package. You should now be able to read the color image, depth map, and command the servo.

The next step will likely be to interpret the data in some form. The Point Cloud Library will be very useful for processing the data. A tutorial on Point Cloud Library is available, although many features are breezed over.

But to begin with PCL, you will need Eigen, a fast vector and matrix math library. (Also used in MRPT)
First download Eigen, and unzip it. Building Eigen is quite straight forward:
cd eigen-eigen-3.0.1
mkdir build
cd build/
cmake ../
sudo make install

PCL has a self installing package, and will install itself to /usr/local/lib/libpcl_*
Here is a simple test program from the tutorial:


#include
#include "pcl/io/pcd_io.h" 
#include "pcl/point_types.h" 

int main (int argc, char** argv) { 
    pcl::PointCloud cloud; 
    //Fill the cloud data 
    cloud.width = 5; 
    cloud.height = 1; 
    cloud.is_dense = false; 
    cloud.points.resize(cloud.width * cloud.height); 
    
    for (size_t i = 0; i < cloud.points.size(); ++i) { 
        cloud.points[i].x = 1024 * rand() / (RAND_MAX + 1.0); 
        cloud.points[i].y = 1024 * rand() / (RAND_MAX + 1.0); 
        cloud.points[i].z = 1024 * rand() / (RAND_MAX + 1.0); 
    } 
    
    pcl::io::savePCDFileASCII("test_pcd.pcd", cloud); 
    std::cerr << "Saved " << cloud.points.size () << " data points to test_pcd.pcd. " << std::endl; 
    
    for (size_t i = 0; i < cloud.points.size(); ++i) 
        std::cerr << "    " << cloud.points[i].x << " " << cloud.points[i].y << " " << cloud.points[i].z << std::endl; 
    
    return(0); 
} 



This can be compiled directly with:
g++ pcd_write.cpp -c -I /usr/local/include/pcl-1.0/ -I /usr/local/include/eigen3/ 
g++ pcd_write.o /usr/local/lib/libpcl_io.dylib 

Finally, attached is a youtube video showing some very impressive visual SLAM surface reconstruction with the Kinect (See KinectFusion project page).


Of course, you will probably want some image stabilization technology to get smooth input videos, so why not try an Owl?


Happy Hacking!

Friday, August 12, 2011

ROS - Robot Operating System

Not so long ago there were a number of robotics packages that provided a variety of functionality of various tasks, such as OpenCV for computer vision, MRPT for mobile robots and ORCOS for general functions. Recently, this has changed as Robot Operating System (ROS) has become the predominant system which has swallowed the functionality of all the other packages.

ROS functionality includes:
Refer to this full list of ROS packages. The functionality encapsulated by ROS is both amazing and disturbing.

I've never been a fan of massive (disorganised!) frameworks that force a work structure onto you, but the robotics world isn't leaving you with much choice anymore. After implementing a graph based information flow architecture in ImprovCV and working closely with DirectShow's graphfilter I'm also keenly aware of the limitations of a graph-based approach to software development. It's not a one-size-fits-all solution, unfortunately ROS leaves little choice.

Thankfully some ROS packages are largely stand alone, so there is still some hope that one day some organisation will put in the effort to streamline and clean up ROS overall.

In any case getting ROS running on Apple Mac OSX 10.6 was not too difficult, however I lost my install notes in an untimely crash, so I'll only give a brief overview. ROS installation relies on MacPorts. I'd been holding off on installing MacPorts as it has historically been unstable, and doesn't play nicely with already installed libraries. MacPorts is now stable and well tested, but it still doesn't play nice with the rest of your system.

To install ROS I followed the ROS diamondback OSX installation instructions, all without any major dramas. I did have to alter the CMakeLists.txt as suggested in the troubleshooting section.

MacPorts gave me a few headaches, some of the errors I encountered were:
  • Error: Target org.macports.install returned: no destroot found

    To solve, use:
    sudo port clean NAME
  • Error: NAME already exists and does not belong to a registered port

    To solve, use:
    sudo port -f activate NAME

Once ROS is all installed you can source setup.sh. Remember to add in any additional packages to ROS_PACKAGE_PATH.

All in all, I wouldn't recommend running ROS on OSX, although the base ROS install works without any major dramas, installing each individual package is a bit of a struggle.

I believe ROS is an appropriate name, it really should be a stand alone operating system. I would strongly advise setting up a separate linux OS dedicated to your ROS install, running ROS ontop of OSX is quite problematic.

Wednesday, August 10, 2011

Communications and Messaging Frameworks

A common problem for any distributed software project is handling inter process communications and messaging. Luckily, a number of frameworks already exist to solve this issue. Robotics projects tend to have a range of requirements, from providing a higher level cognitive architecture framework (typically a set of interacting components or nodes) to low-level realtime communications (e.g. CANbus, RT ethernet).

Some robotics packages already provide a particular architecture, such as ImprovCV and ROS which are both filter based, or MRPT which provides some IPC primitives. However, this problem is hardly limited to robotics and many real time distributed networking/processing systems have similar communications requirements. (e.g. computer games, or traffic management systems / plant control systems).

The communications frameworks can be broadly grouped into three categories:
  1. Integrated middlewear for a larger package
  2. (hard) Real time control orientated frameworks
  3. (soft) Real time messaging libraries
From a games perspective, the people at second life have a nice evaluation of several messaging systems. Beginning with soft real time messaging, some interesting libraries include:
  • Google protocol buffers, for serialisation. I mention it, because it is popular.
  • MessagePack is a language agnostic message serialisation system similar to Google protocol buffers - just faster and better.
  • CORBA allows objects to be sent around a network, often used in business applications. It is an open standard and language agnostic as it includes an Interface Definition Language to define components.
  • Microsoft DCOM / .NET remote and its many variants are similar to CORBA, just the Microsoft variant.
  • Apache ActiveMQ another enterprise orientated message broker with support for a large number of scripting languages (Python, PHP, Ruby, Perl, etc).
  • RakNet manages distributed objects, primarily targeted towards computer games (used in Unity).

  • Open Torque Networking Library has its origins in the Torque game engine designed for massively multiplayer games.

  • YAMI4 is a fairly advanced but easy to use peer to peer messaging framework for control systems that is easy to extend to a pub/sub system. POSIX, Windows, Java, C++, well worth a look.

  • 0MQ, zero MQ is a messaging system that is socket based, and also has pub/sub extensions.

  • Socket.IO hardly qualifies for a framework, but is a great enabler for realtime HTML communications, so it certainly deserves a mention.

there are many more networking libraries available, but these are the ones that stand out for this category. For hard real-time control tasks the choices are a bit more limited, however libraries of note are:
  • Real time CORBA, is CORBA modified to be a bit more lightweight and able to support real time requirements. TAO ACE is worth looking into if your following this path.
  • Data Distribution Service (DDS) is similar to Real time CORBA, however it has the advantage of lessons learnt from CORBA and provides simpler fine grained control over quality of service and intelligent networking. RTI DDS is the framework we used for the WAMbot TARDEC/DSTO MAGIC 2010 project. OpenDDS is an open source implementation.
  • Real time data base (RTDB) is an open source framework that provides a lower-level communications interface for C/C++ over Xenomai. This is the framework we used for the Team Annieway entry to the DARPA Urban Challenge.
  • LCM is a socket-based IPC library used by Team Michigan for MAGIC 2010.
  • IPC a low level inter process communications system for embedded systems used by NASA.
  • Dynamic Data eXchange (DDX) is a large open source distributed software architecture developed by CSIRO and has been previously integrated with ROS.
Finally, there are a number of application specific frameworks, for Robotics this includes ROS, MS Robot Studio, Player/Stage, JAUS and more general packages such as Labview and Matlab. I think it is prudent for every software developer to be familiar with at least three tiers of communications frameworks, from large scale frameworks (e.g. CORBA) to light-weight frameworks (e.g. RackNet) to low-level frameworks (e.g. Socket.IO). Ideally, one for each key application area (e.g. Hard real time, Soft real time, Business applications, Web).

Sunday, June 05, 2011

Catchup Post: Robotics and Physical Simulations

A number of robotics related bits of interest from the last few months:
Finally Heat-1 the space rocket build by danish amateurs Copenhagen Suborbitals successfully launched. (TED talk here). See the video below.

Thursday, March 31, 2011

March update

I didn't want to have a month without a post, so here are some links for the month:
The pick of the month has to be the new Festo production, the SmartBird ornithopter. (Official site has a PDF spec-sheet). Video below:

Thursday, February 24, 2011

WAMBot on ABC Catalyst

ABC Catalyst did a segment on the MAGIC 2010 competition, you can watch the Catalyst RobotWars MAGIC2010 segment on their website.

A more in-depth explanation of the problems we faced: The robots were stored overnight on site in a tent, and unfortunately the next day a lot of moisture had gathered in the electronics. Since we had used cheap GPS units, a number of them then failed on the day. The software we had written was not robust enough to cope with the faulty units and so the robots would think they were in an incorrect location. As the fleet of robots could not agree on their locations the map data was incomplete, and subsequently the robots were difficult to navigate.

Other software bugs, short battery life and communications drops also hindered the performance.

Next time: More testing, and focus on robustness and fallback options, and buy equipment with an IP rating of 32 or more.

A short clip of the interview is on youtube:

Thursday, December 23, 2010

Underground

Few updates this month because I've been underground. And I don't mean I've joined some kind of strange sub-culture or gone in hiding, I mean literally 1.5km vertically underground to commission a Transmin automated Rockbreaker (there's not much internet down there!). A few things I learnt / experienced about hard-rock underground mines (most of which are very obvious):
  • Block caving is insane. The idea is to create an extraction level under the ore body, then continuously blast the ore into smaller rocks that can be processed, leaving a massive growing "sink" hole on the surface. You also need a very very very long conveyor belt.
  • The tunnels are quite large, as some very large vehicles need to drive through the mine.
  • All the tunnels have a 'clean' look to them, largely thanks to the use of shotcrete (i.e. concrete reinforcement around the tunnel rock).
  • It takes 45 minutes to drive down to a depth of 1.5km
  • When driving, give way to larger vehicles, or vehicles with explosives.
  • Its difficult to move around quickly, you will easily break a sweat in no time.
  • Batteries weigh a lot when you need to carry them around on a belt all day
  • You use a lot more oxygen if you panic, or move around a lot
  • There is a complex system of ventilation channels with really really big fans providing all the air
  • It is really easy to get lost in a large underground mine
  • There is an underground dining area. It is called the 'crib'. I don't know why.
  • If anything bad happens, you don't have much of a chance, especially if you are not aware of all the intimate details of how the mine operates. (ie: haven't been there for more than a year)
  • Explosions are loud
  • Ground movement is bad and very very loud
  • There is an awful lot of dust
All in all an interesting experience, but all the dust certainly didn't do any good for my eyes or my lungs. I've been coughing a lot even after being out of the mine for a week as the very fine dust just gets everywhere.