Saturday, April 1, 2017

A 360° view on auto-follow-drones

Problem is always the same: You want to take videos of how you ride but for some reason nobody is willing to film you. They want to ride as well.

With the advent of drones aerial videography comes into the hands of hobby users, but from a practical point of view it is even worse. Now you need a friend who can fly a drone in order to film you.

The solution to both problems at once are auto-follow drones. You have a camera following you and you can get aerial shots as well. Perfect solution in theory.

Battery endurance


The first and most obvious problem is the battery endurance. I guess nobody expects a drone to follow you all day. On the other hands, the typical flight time is 20 to 30 minutes. At first that might sound like little but imagine your sport. Okay, for filming a 90 minute soccer game, that won't work. But neither is this the sport you use drones for. We rather talk about downhill biking, skiing - or in my case - snowboarding. And here a typical cycle is a 10 minute ascend and then a 2 minute run. As you would film your downhill runs only, you can get quite far with 20 minutes out of flight time.
The other thing to consider is that an aerial video is an interesting perspective and the such, granted. But watching minutes after minutes of the same thing and from the same angle gets boring quickly.
Therefore you will adjust the follow distance, the position to front/side/rear or use advanced flight modes like orbiting you. But even with this, there is only so much you can use in a regular 3 minute youtube video.
Don't get me wrong, would love to have hours of flight time. And with my DJI Inspire I carry a lot of batteries with me. But that is because you do scenic shots, rehearse the same take many times until it is perfect and things like that. You can go down the same slope only so much in one day.
Hence my point, a flight time of 20 minutes per battery is okay.

Conditions apply


The by far biggest problem is how much it needs to take drone videos. Skiing/Snowboarding is by far the worst for that.

  1. Sun - A video in bad weather looks just grey and dark and boring. You need blue sky and sun.
  2. No Wind - With gusts of 4Bft or more you can forget flying.
  3. Snow conditions - You want to show off instead, not watch how you were skidding on ice.
  4. No people for security reasons - So filming on the slope on weekends or during the day is impossible, you can hope for the first two to three runs, if that.
  5. Takeoff and landing spot - In the mountains you typically cannot land wherever you want in case of an emergency (spot landing), the drone will return to the start point. Hence you need a horizontal spot (3x3 meter to allow for GPS errors) as emergency landing area.
How likely is it to find all requirements met? As said, my use case is probably the worst. Backcountry skiing you eliminate requirement 4) essentially, take downhill mountain biking and 3) will not play a big role in addition. I just want to raise the awareness and you to consider these factors upfront.



The three types of auto-follow-drones


Looking at the market, DJI is relying on optical image recognition for all its drones in order to follow you, Hexo+ is using your cell phone's GPS signal and Airdog & Staaker require you to wear a GPS handheld device.
Here a first differentiation of what auto-follow means is in order. Optical image recognition requires the color, shape and movement to be similar all the time. One example could be you skiing straight downhill. Will not be much of a problem to track based on the camera image alone. At the end that is what every pilot does all the time. He looks at the live feed and tries to keep you in the center. But this technology is not perfect. A short sun flare in the camera, you going into a shadow, you passing by somebody else,... all can throw off the algorithms. And the result is that the drone stops following you immediately until you pilot it manually. Or worse, it decided to want a new owner and follows somebody else?
As result you constantly need to check if the drone is following you.
In reality this follow technology is good for occasional strolls along the beach but not for sports action.
Actually, all those drone do at the end is simplify the pilots life by operating the camera but you need a drone pilot still.
Note: DJI drones do not descent while in active track mode


Following the GPS signal of the cell phone is another theoretically elegant solution to the auto-follow problem but again not for action sports.
One thing this enables for sure is to recapture you. If the signal is lost for a few seconds, the drone has to stop. Just as if the rider is behind a tree for a second and cannot be reacquired by the camera in case of optical following. But as soon as the signal is back, the drone can follow you again, regardless of where you are now. Hence a GPS based follow is a practical requirement fro me.
The problem is the accuracy. A smartphone sends out one GPS position per second and its GPS signal is not the most accurate one. So say you go down a hill as slow as 10m/s, the GPS signal of your phone is 3m off and the one in the drone another 3m. Either you zoom out so much that you are a black dot in the landscape or you will be in the picture of the camera for moments only.
Also smartphones do have short range bluetooth chips only.
Then there are all kinds of practical problems, you need to get to your phone, remove the gloves,... even for the most basic operations like making it land.

In contrast the custom wrist mounted GPS controllers like the Airleash used by the Airdog reads as

  • 10 GPS positions per second
  • Overlaid with acceleration sensor based Inertial Measurement Unit for more accurate position information
  • Barometric sensor for height precision of better than 0.1 meter
  • Convenience of buttons and LCD readable in the sun
  • Bluetooth class 1

So frankly, that is the only real option today.


Longterm the combination of optical and GPS would be the optimal solution. I have no guess how long that will take to get reality, though.


Obstacle avoidance

This is an often stated requirement which collides with reality. The by far leading drones including obstacle avoidance are the recent DJI drones: Mavic, Inspire 2 and Phantom 4 Pro. But even for these the obstacle avoidance is a pilot support. You can still crash these drones and it does not take much for that. For a real auto-follow drone you have to rely on these features. Period.
Apart of its shortcomings, there is are the system inherent issues as well. As all is based on optical sensors (plus short range ultrasound sensors facing downwards for landing), you need contrast. A white wall, looking directly into the sun,... all could throw off the obstacle avoidance.
Therefore this feature is really great for indoor flying, when there is no GPS signal and lots of stuff in the way. And you fly at low speeds. But for outdoors it is a help at most and that is too little for an auto-follow drone.
If there is a 3D LIDAR sometimes in the future, that will be the first time obstacle avoidance can be trusted outdoors enough. (Although even than a lift cable will probably be recognized too late)

Hence my take for the time being is, you do not need obstacle avoidance you cannot trust, rather flay where there are no obstacles. Check the surroundings first.

A completely different story is looking downwards. As most of our sports will involve altitude changes, it will be the most common error to hit the ground. Initially Airdog did not have any protection for that and the company learned it the hard way. So the delayed the Airdog by another 6 months just to add a long range, high precision LIDAR (Laser based distance measuring) facing downwards. As soon as the clearance is below a threshold, Airdog will start climbing right away to protect itself. After a couple of flights that saved me twice already.
In addition the LIDAR can be used to support the landing procedure, another difficult task for.

One thing the DJI birds have but not the Airdog is a down-facing camera. This is used to support hovering in the exact same position and to improve the landing precision. Airdog solely relies on the GPS position and therefore can land 1-2m away from the take off point easily. With vision support it could have landen on a portable heli pad instead. Would help a lot.


Legal situation

I know the legal situation for Europe and US. In both cases you have to fly in Line-of-sight, meaning you have to watch at your drone all the time and you have to have your fingers on the controller at any time. Yes, it does mean that and nothing else.
So in that sense only the DJI drone is legal, as there all the features are used to support the pilot and not replace him temporarily. The pilot can take over control at any time.
Hexo+, Airdog, Staaker and all other auto-follow drones heavier than 250g, so essentially all drones except toys, are legal to fly when you keep your hands on the controller and keep looking at the copter permanently and you are prepared to take over at any time. It is possible to fly that way but it is not the intended way. We want to film ourselves without the need of another pilot.
To make them legal and take advantage of their autonomous capabilities, we would need a second controller handed over to a friend who acts as the supervising pilot. In worst case all he does is pressing the pause button to stop following you. But that is the law.

Another aspect is flying in the public. By law you can fly only in areas where nobody else is within 50m-100m of the drone, except people required for the video. And no buildings either. (Europe)
Such breath taking drone shots Casey Neistat created in the past are not only illegal but could cause real harm to bystanders. Just imagine the drone falls out of the sky hitting the front of a car. You will be scared and panic.
Even he did accept that recently.
https://youtu.be/eoV82ryz_BI?t=2m5s

So the legal situation limits you even further.
In Austria, where I am flying, the situation is a little bit worse. For example you cannot fly when other people are within 50m. Understandable. And not on Sundays, because you might disturb the non-existing people. (From 50m away you can barely see a drone of this size nor hear it.)


Summary of me and my #Airdog


Airdog is the only product which meets my needs. It is accurate enough, it has obstacle avoidance facing down, it is fast and it is portable.
One thing that is missing is the ability to have a standby pilot in order to meet the law requirements.
You need lots of prep work.
But when all comes together, the shots you get are worth every penny.

To give you an impression, check out my youtube video of this run. It is out of the cam and taken from too far away, but it shows all the things that could have gone wrong. If I would have flown with another drone...

  • It would have crashed into the mountain as there was a hump in the line of the flight path.
  • It would have landed due to an out-of-battery situation in the middle of nowhere, with no place for a safe landing.
  • It would not have been able to keep up with me so easily




And another video to show you the limits. I make so sharp 180° turns and Airdog is so close, it tries to stay relative to my current position and barely manages. Which is good from one perspective - it obviously knows my position precisely - but bad from another point of view - I can make sharper turns cutting deep into the snow than a drone can manage in air.



Totally different story when making less radical turns.



Post a Comment