---
author:
   email: [email protected]
   image: https://petermolnar.net/favicon.jpg
   name: Peter Molnar
   url: https://petermolnar.net
copies:
- http://web.archive.org/web/20180928133438/https://petermolnar.net/location-tracking-without-server/
lang: en
published: '2018-09-27T11:05:00+01:00'
summary: My pipeline of tracking where I've been using and Android phone, Backitude,
   python, and cron
tags:
- android
title: GPS tracking without a server

---

Nearly all self-hosted location tracking Android applications are based
on server-client architecture: the one on the phone collects only a
small points, if not only one, and sends it to a configured server.
Traccar[^1], Owntracks[^2], etc.

While this setup is useful, it doesn't fit in my static, unless it
hurts[^3] approach, and it needs data connectivity, which can be tricky
during abroad trips. The rare occasions in rural Scotland and Wales
tought me data connectivity is not omnipresent at all.

There used to be a magnificent little location tracker, which, besides
the server-client approach, could store the location data in CSV and KML
files locally: Backitude[^4]. The program is gone from Play store, I
have no idea, why, but I have a copy of the last APK of it[^5].

My flow is the following:

-   Backitude saves the CSV files
-   Syncthing[^6] syncs the phone and the laptop
-   the laptop has a Python script that imports the CSV into SQLite to
   eliminate duplicates
-   the same script queries against Bing to get altitude information for
   missing altitudes
-   as a final step, the script exports daily GPX files
-   on the laptop, GpsPrune helps me visualize and measure trips

## Backitude configuration

These are the modified setting properties:

-   Enable backitude: yes
-   Settings
   -   Standard Mode Settings
       -   Time Interval Selection: 1 minute
       -   Location Polling Timeout: 5 minutes
       -   Display update message: no
   -   Wifi Mode Settings
       -   Wi-Fi Mode Enabled: yes
       -   Time Interval Options: 1 hour
       -   Location Polling Timeout: 5 minutes
   -   Update Settings
       -   Minimum Change in Distance: 10 meters
   -   Accuracy Settings
       -   Minimum GPS accuracy: 12 meters
       -   Minimum Wi-Fi accuracy: 20 meters
   -   Internal Memory Storage Options
       -   KML and CSV
   -   Display Failure Notifications: no

I have an exported preferences file available[^7].

## Syncthing

The syncthing configuration is optional; it could be simple done by
manual transfers from the phone. It's also not the most simple thing to
do, so I'll let the Syncting Documentation[^8] take care of describing
the how-tos.

## Python script

Before jumping to the script, there are 3 Python modules it needs:

``` {.bash}
pip3 install --user arrow gpxpy requests
```

And the script itself - please replace the `INBASE`, `OUTBASE`, and
`BINGKEY` properties. To get a Bing key, visit Bing[^9].

``` {.python}
import os
import sqlite3
import csv
import glob
import arrow
import re
import gpxpy.gpx
import requests

INBASE="/path/to/your/syncthing/gps/files"
OUTBASE="/path/for/sqlite/and/gpx/output"
BINGKEY="get a bing maps key and insert it here"

def parse(row):
   DATE = re.compile(
       r'^(?P<year>[0-9]{4})-(?P<month>[0-9]{2})-(?P<day>[0-9]{2})T'
       r'(?P<time>[0-9]{2}:[0-9]{2}:[0-9]{2})\.(?P<subsec>[0-9]{3})Z$'
   )

   lat = row[0]
   lon = row[1]
   acc = row[2]
   alt = row[3]
   match = DATE.match(row[4])
   # in theory, arrow should have been able to parse the date, but I couldn't get
   # it working
   epoch = arrow.get("%s-%s-%s %s %s" % (
       match.group('year'),
       match.group('month'),
       match.group('day'),
       match.group('time'),
       match.group('subsec')
   ), 'YYYY-MM-DD hh:mm:ss SSS').timestamp
   return(epoch,lat,lon,alt,acc)

def exists(db, epoch, lat, lon):
   return db.execute('''
       SELECT
           *
       FROM
           data
       WHERE
           epoch = ?
       AND
           latitude = ?
       AND
           longitude = ?
   ''', (epoch, lat, lon)).fetchone()

def ins(db, epoch,lat,lon,alt,acc):
   if exists(db, epoch, lat, lon):
       return
   print('inserting data point with epoch %d' % (epoch))
   db.execute('''INSERT INTO data (epoch, latitude, longitude, altitude, accuracy) VALUES (?,?,?,?,?);''', (
       epoch,
       lat,
       lon,
       alt,
       acc
   ))


if __name__ == '__main__':
   db = sqlite3.connect(os.path.join(OUTBASE, 'location-log.sqlite'))
   db.execute('PRAGMA auto_vacuum = INCREMENTAL;')
   db.execute('PRAGMA journal_mode = MEMORY;')
   db.execute('PRAGMA temp_store = MEMORY;')
   db.execute('PRAGMA locking_mode = NORMAL;')
   db.execute('PRAGMA synchronous = FULL;')
   db.execute('PRAGMA encoding = "UTF-8";')

   files = glob.glob(os.path.join(INBASE, '*.csv'))
   for logfile in files:
       with open(logfile) as csvfile:
           try:
               reader = csv.reader(csvfile)
           except Exception as e:
               print('failed to open CSV reader for file: %s; %s' % (logfile, e))
               continue
           # skip the first row, that's headers
           headers = next(reader, None)
           for row in reader:
               epoch,lat,lon,alt,acc = parse(row)
               ins(db,epoch,lat,lon,alt,acc)
       # there's no need to commit per line, per file should be safe enough
       db.commit()

   db.execute('PRAGMA auto_vacuum;')

   results = db.execute('''
       SELECT
           *
       FROM
           data
       ORDER BY epoch ASC''').fetchall()
   prevdate = None
   gpx = gpxpy.gpx.GPX()

   for epoch, lat, lon, alt, acc in results:
       # in case you know your altitude might actually be valid with negative
       # values you may want to remove the -10
       if alt == 'NULL' or alt < -10:
           url = "http://dev.virtualearth.net/REST/v1/Elevation/List?points=%s,%s&key=%s" % (
               lat,
               lon,
               BINGKEY
           )
           bing = requests.get(url).json()
           # gotta love enterprise API endpoints
           if not bing or \
               'resourceSets' not in bing or \
               not len(bing['resourceSets']) or \
               'resources' not in bing['resourceSets'][0] or \
               not len(bing['resourceSets'][0]) or \
               'elevations' not in bing['resourceSets'][0]['resources'][0] or \
               not bing['resourceSets'][0]['resources'][0]['elevations']:
               alt = 0
           else:
               alt = float(bing['resourceSets'][0]['resources'][0]['elevations'][0])
               print('got altitude from bing: %s for %s,%s' % (alt,lat,lon))
               db.execute('''
                   UPDATE
                       data
                   SET
                       altitude = ?
                   WHERE
                       epoch = ?
                   AND
                       latitude = ?
                   AND
                       longitude = ?
                   LIMIT 1
               ''',(alt, epoch, lat, lon))
               db.commit()
           del(bing)
           del(url)
       date = arrow.get(epoch).format('YYYY-MM-DD')
       if not prevdate or prevdate != date:
           # write previous out
           gpxfile = os.path.join(OUTBASE, "%s.gpx" % (date))
           with open(gpxfile, 'wt') as f:
               f.write(gpx.to_xml())
               print('created file: %s' % gpxfile)

           # create new
           gpx = gpxpy.gpx.GPX()
           prevdate = date

           # Create first track in our GPX:
           gpx_track = gpxpy.gpx.GPXTrack()
           gpx.tracks.append(gpx_track)

           # Create first segment in our GPX track:
           gpx_segment = gpxpy.gpx.GPXTrackSegment()
           gpx_track.segments.append(gpx_segment)

       # Create points:
       gpx_segment.points.append(
           gpxpy.gpx.GPXTrackPoint(
               lat,
               lon,
               elevation=alt,
               time=arrow.get(epoch).datetime
           )
       )

   db.close()
```

Once this is done, the `OUTBASE` directory will be populated by `.gpx`
files, one per day.

## GpsPrune

GpsPrune is a desktop, QT based GPX track visualizer. It needs data
connectivity to have nice maps in the background, but it can do a lot of
funky things, including editing GPX tracks.

``` {.bash}
sudo apt install gpsprune
```

**Keep it in mind that the export script overwrites the GPX files, so
the data needs to be fixed in the SQLite database.**

This is an example screenshot of GpsPrune, about our 2 day walk down
from Mount Emei and it's endless stairs:


Happy tracking!

[^1]: <https://www.traccar.org/>

[^2]: <https://owntracks.org/>

[^3]: <https://indieweb.org/manual_until_it_hurts>

[^4]: <http://www.gpsies.com/backitude.do>

[^5]: [gaugler.backitude.apk](gaugler.backitude.apk)

[^6]: <https://syncthing.net/>

[^7]: [backitude.prefs](backitude.prefs)

[^8]: <https://docs.syncthing.net/intro/getting-started.html>

[^9]: <https://msdn.microsoft.com/en-us/library/ff428642>

You are viewing proxied material from petermolnar.net. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.