My Own Mapserver: Unterschied zwischen den Versionen

Aus DL8RDS Wiki
Wechseln zu: Navigation, Suche
(→‎Get the Mapnik style)
(→‎Method 3: generate_tiles.py with Mapnik)
 
(31 dazwischenliegende Versionen desselben Benutzers werden nicht angezeigt)
Zeile 7: Zeile 7:
 
* http://44.225.36.250/osm/slippymap.html
 
* http://44.225.36.250/osm/slippymap.html
  
−
== Implementation ==
+
== Some basics ==
 +
 
 +
* http://wiki.openstreetmap.org/wiki/Slippy_Map
 +
* http://wiki.openstreetmap.org/wiki/Tiles
 +
 
 +
== Mapserver Setup ==
  
 
I am replicating this report:
 
I am replicating this report:
  
 
* http://blog.loco-toys.de/wordpress/2017/04/eigener-mapserver-mit-openstreetmap-mapnik-und-openlayers/
 
* http://blog.loco-toys.de/wordpress/2017/04/eigener-mapserver-mit-openstreetmap-mapnik-und-openlayers/
 +
 +
Other interesting guides:
 +
 +
* https://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/
  
 
=== Preparations ===
 
=== Preparations ===
Zeile 198: Zeile 207:
 
Modify the file '''datasource-settings.xml.inc''' like this:
 
Modify the file '''datasource-settings.xml.inc''' like this:
  
 +
<pre>
 
<!--
 
<!--
 
Settings for your postgres setup.
 
Settings for your postgres setup.
Zeile 206: Zeile 216:
 
<Parameter name="type">postgis</Parameter>
 
<Parameter name="type">postgis</Parameter>
 
<!-- <Parameter name="password">%(password)s</Parameter> -->
 
<!-- <Parameter name="password">%(password)s</Parameter> -->
−
<!-- <Parameter name="host">%(host)s</Parameter> -->
+
<!-- <Parameter name="host"></Parameter> -->
−
<!-- <Parameter name="port">%(port)s</Parameter> -->
+
<!-- <Parameter name="port">5432</Parameter> -->
−
<!-- <Parameter name="user">%(user)s</Parameter> -->
+
<!-- <Parameter name="user">gis</Parameter> -->
 
<Parameter name="dbname">gis</Parameter>
 
<Parameter name="dbname">gis</Parameter>
 
<!-- this should be 'false' if you are manually providing the 'extent' -->
 
<!-- this should be 'false' if you are manually providing the 'extent' -->
Zeile 215: Zeile 225:
 
<!-- providing this speeds up Mapnik database queries -->
 
<!-- providing this speeds up Mapnik database queries -->
 
<Parameter name="extent">-20037508,-19929239,20037508,19929239</Parameter>
 
<Parameter name="extent">-20037508,-19929239,20037508,19929239</Parameter>
 +
</pre>
 +
 +
Finally copy the mapnik-style directory over to its default location at '''/etc/mapnik-osm-data''':
 +
 +
mv ~./src/mapnik-styles /etc/mapnik-osm-data
  
 
=== Install and Configure mod_tile and renderd ===
 
=== Install and Configure mod_tile and renderd ===
Zeile 220: Zeile 235:
 
There are several recommendations to simply build and use mod_tile from source. I decided to generate a debian package first and then install the package properly. I followed this recommendation:
 
There are several recommendations to simply build and use mod_tile from source. I decided to generate a debian package first and then install the package properly. I followed this recommendation:
  
−
  http://wiki.openstreetmap.org/wiki/Mod_tile
+
  https://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/
 +
 
 +
After you have installed mod_tile, there should be the following files:
 +
 
 +
<pre>
 +
root@mapserver:/home/dl8rds/src# dpkg -L libapache2-mod-tile
 +
/.
 +
/var
 +
/var/www
 +
/var/www/osm
 +
/var/www/osm/slippymap.html
 +
/usr
 +
/usr/bin
 +
/usr/bin/openstreetmap-tiles-update-expire
 +
/usr/bin/osmosis-db_replag
 +
/usr/share
 +
/usr/share/munin
 +
/usr/share/munin/plugins
 +
/usr/share/munin/plugins/mod_tile_fresh
 +
/usr/share/munin/plugins/mod_tile_latency
 +
/usr/share/munin/plugins/mod_tile_zoom
 +
/usr/share/munin/plugins/mod_tile_response
 +
/usr/share/doc
 +
/usr/share/doc/libapache2-mod-tile
 +
/usr/share/doc/libapache2-mod-tile/copyright
 +
/usr/share/doc/libapache2-mod-tile/changelog.Debian.gz
 +
/usr/share/doc/libapache2-mod-tile/readme.txt.gz
 +
/usr/lib
 +
/usr/lib/apache2
 +
/usr/lib/apache2/modules
 +
/usr/lib/apache2/modules/mod_tile.so
 +
/etc
 +
/etc/apache2
 +
/etc/apache2/mods-available
 +
/etc/apache2/mods-available/tile.load
 +
/etc/apache2/sites-available
 +
/etc/apache2/sites-available/tileserver_site.conf
 +
</pre>
 +
 
 +
And the same with renderd:
 +
 
 +
<pre>
 +
root@mapserver:/home/dl8rds/src# dpkg -L renderd
 +
/.
 +
/usr
 +
/usr/bin
 +
/usr/bin/render_old
 +
/usr/bin/render_expired
 +
/usr/bin/renderd
 +
/usr/bin/render_speedtest
 +
/usr/bin/render_list
 +
/usr/share
 +
/usr/share/munin
 +
/usr/share/munin/plugins
 +
/usr/share/munin/plugins/renderd_processed
 +
/usr/share/munin/plugins/renderd_zoom_time
 +
/usr/share/munin/plugins/renderd_zoom
 +
/usr/share/munin/plugins/renderd_queue
 +
/usr/share/munin/plugins/renderd_queue_time
 +
/usr/share/doc
 +
/usr/share/doc/renderd
 +
/usr/share/doc/renderd/copyright
 +
/usr/share/doc/renderd/changelog.Debian.gz
 +
/usr/share/doc/renderd/readme.txt.gz
 +
/usr/share/man
 +
/usr/share/man/man8
 +
/usr/share/man/man8/renderd.8.gz
 +
/usr/share/man/man1
 +
/usr/share/man/man1/render_expired.1.gz
 +
/usr/share/man/man1/render_speedtest.1.gz
 +
/usr/share/man/man1/render_old.1.gz
 +
/usr/share/man/man1/render_list.1.gz
 +
/usr/lib
 +
/usr/lib/libiniparser.so.3.0.0
 +
/usr/lib/libiniparser.la
 +
/usr/lib/libiniparser.a
 +
/etc
 +
/etc/renderd.conf
 +
/etc/init.d
 +
/etc/init.d/renderd
 +
/usr/lib/libiniparser.so.3
 +
/usr/lib/libiniparser.so
 +
</pre>
 +
 
 +
Now edit the file '''/etc/renderd.conf''' like this:
 +
 
 +
[renderd]
 +
stats_file=/var/run/renderd/renderd.stats
 +
socketname=/var/run/renderd/renderd.sock
 +
num_threads=4
 +
tile_dir=/var/www/tiles
 +
 +
[mapnik]
 +
plugins_dir=/usr/lib/mapnik/3.0/input
 +
font_dir=/usr/share/fonts
 +
font_dir_recurse=true
 +
 +
[osm]
 +
HOST=localhost
 +
TILESIZE=256
 +
URI=/tiles/osm/
 +
XML=/etc/mapnik-osm-data/osm.xml
 +
DESCRIPTION=This is the standard osm mapnik style
 +
HOST=mapserver.db0hsr.apr.org
 +
 
 +
Finally ensure that the modules are used by Apache:
 +
 
 +
$ sudo sh -c 'echo "LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so" > /etc/apache2/mods-available/tile.load'
 +
 
 +
Activate it without an Apache restart:
 +
 
 +
$ sudo a2enmod tile
 +
 
 +
Now modify the website definition. Decide what to do with the default website config '''000-default.conf'''. I simply deleted it.
 +
 
 +
$ sudo nano /etc/apache2/sites-available/tileserver_site.conf
 +
 
 +
Make sure that the file looks like this:
 +
 
 +
<pre>
 +
<VirtualHost *:80>
 +
    ServerAdmin dl8rds@darc.de
 +
    ServerName mapserver.db0hsr.ampr.org
 +
    DocumentRoot /var/www
 +
    LogLevel info
 +
    AddTileConfig /tiles/ osm
 +
    LoadTileConfigFile /etc/renderd.conf
 +
    ModTileRequestTimeout 3
 +
    ModTileMissingRequestTimeout 10
 +
    ModTileMaxLoadOld 2
 +
    ModTileMaxLoadMissing 5
 +
    ModTileRenderdSocketName /var/run/renderd/renderd.sock
 +
    ModTileCacheDurationMax 604800
 +
    ModTileCacheDurationDirty 900
 +
    ModTileCacheDurationMinimum 10800
 +
    ModTileCacheDurationMediumZoom 13 86400
 +
    ModTileCacheDurationLowZoom 9 518400
 +
    ModTileCacheLastModifiedFactor 0.20
 +
    ModTileEnableTileThrottling Off
 +
    ModTileEnableTileThrottlingXForward 0
 +
    ModTileThrottlingTiles 10000 1
 +
    ModTileThrottlingRenders 128 0.2
 +
        <Directory />
 +
Options FollowSymLinks
 +
AllowOverride None
 +
</Directory>
 +
<Directory /var/www/>
 +
Options Indexes FollowSymLinks MultiViews
 +
AllowOverride None
 +
Order allow,deny
 +
allow from all
 +
</Directory>
 +
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
 +
<Directory "/usr/lib/cgi-bin">
 +
AllowOverride None
 +
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
 +
Order allow,deny
 +
Allow from all
 +
</Directory>
 +
    Alias /doc/ "/usr/share/doc/"
 +
    <Directory "/usr/share/doc/">
 +
        Options Indexes MultiViews FollowSymLinks
 +
        AllowOverride None
 +
        Order deny,allow
 +
        Deny from all
 +
        Allow from 127.0.0.0/255.0.0.0 ::1/128
 +
    </Directory>
 +
</VirtualHost>
 +
</pre>
 +
 
 +
Finally restart Apache:
 +
 
 +
$ sudo apachectl -t
 +
 
 +
and it should say something like this:
 +
 
 +
Syntax OK
 +
 
 +
Oh yes, don't forget to create the runtime directory for renderd:
 +
 
 +
$ sudo mkdir /var/run/renderd /var/lib/mod_tile
 +
$ sudo chown www-data:www-data /var/run/renderd /var/lib/mod_tile
 +
 
 +
=== Download Map Data ===
 +
 
 +
Get the dataset for Germany. A company called Geofabrik is providing these files:
 +
 
 +
http://download.geofabrik.de/europe/germany.html
 +
 
 +
Now download:
 +
 
 +
$ mkdir ~/osm && cd ~/osm
 +
wget http://download.geofabrik.de/europe/germany-latest.osm.pbf
 +
 
 +
=== Prepare the Database ===
 +
 
 +
There are packages for Ubuntu. You can compile the program yourself, too. I prefer the prefabricated package.
 +
 
 +
$ sudo apt-get install osm2pgsql
  
−
First provide the prerequisites:
+
Then prepare the database:
  
−
  apt-get install libmapnik3.0 libmapnik-dev mapnik-utils python-mapnik autoconf apache2-dev
+
  $ sudo -u postgres createuser gis
 +
$ sudo -u postgres createdb -E UTF8 -O gis gis
 +
$ sudo -u postgres createlang plpgsql gis
 +
 
 +
Edit the file '''/etc/postgresql/9.5/main/pg_hba.conf''':
 +
 
 +
# "local" is for Unix domain socket connections only
 +
local  all        all                              trust
 +
local  gis        gis                              trust
 +
 
 +
There is a script that also installs postgis:
 +
 
 +
/usr/share/doc/osm2pgsql/examples/install-postgis-osm-db.sh
 +
 
 +
Unfortunately it does not work correctly. So let's do it by hand:
 +
 
 +
$ sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'
 +
 
 +
Correct table ownerships:
 +
 
 +
$ echo 'ALTER TABLE geometry_columns OWNER TO gis; ALTER TABLE spatial_ref_sys OWNER TO gis;' | sudo -u postgres psql gis
 +
 
 +
And then import the database:
 +
 
 +
$ osm2pgsql --create --database gis --username gis --slim --cache 4096 --hstore /data/germany-latest.osm.pbf
 +
 
 +
Now wait for a couple of days.
 +
 
 +
=== Start everything ===
 +
 
 +
Make sure you have the output directory hierarchy:
 +
 
 +
$ mkdir /var/www/tiles/osm
 +
$ chown -R www-data:www-data /var/www/tiles/osm
 +
 
 +
Simply make sure all the services are up and running:
 +
 
 +
$ sudo service renderd start
 +
$ sudo systemctl restart apache2
 +
 
 +
=== Precompute Tiles ===
 +
 
 +
First of all, let me state that I have simply decided to base all my maps on Mapnik in order to keep the prcess comparable and simple. It works, and never change a winning team. Not for now.
 +
 
 +
==== Method 1: Tirex with Mapnik -- Abandoned. ====
 +
 
 +
This could be done with Tirex, but I still have some problems with it. So let's check render_list...
 +
 
 +
* http://wiki.openstreetmap.org/wiki/DE:HowTo_Mapnik_%26_Tirex
 +
 
 +
==== Method 2: render_list with Mapnik ====
 +
 
 +
This is my render_list invocation:
 +
 
 +
root@mapserver:~# render_list -v -n 6 -all --socket=/var/run/renderd/renderd.sock --min-zoom=6 --max-zoom=15 -x 2112 -X 2208 -y 2624 -Y 2880
 +
 
 +
... but I'm still playing around with it. Our machine is powerful, but it's not a supercomputer. :-) Here is some good advice (in German):
 +
 
 +
https://forum.openstreetmap.org/viewtopic.php?id=20800
 +
 
 +
==== Method 3: generate_tiles.py with Mapnik ====
 +
 
 +
Here is the decription how to do that: https://sites.google.com/site/nitinpasumarthy/blog/createyourowntileserverandmapclient
 +
 
 +
This is the region I'm interested in: http://www.openstreetmap.org/export#map=7/48.756/12.398&layers=C
 +
 
 +
Here is my python file: /usr/share/osm-mapnik/generate_tiles.py
 +
 
 +
And I see that it can be modified via environment variables, notably:
 +
 
 +
* MAPNIK_MAP_FILE, which is /etc/mapnik-osm-data/osm.xml
 +
* MAPNIK_TILE_DIR, which is /var/www/tiles/osm
 +
 
 +
and this section of the file is also interesting:
 +
 
 +
<pre>
 +
    bbox = (-180.0,-90.0, 180.0,90.0)
 +
 
 +
    render_tiles(bbox, mapfile, tile_dir, 0, 5, "World")
 +
 
 +
    minZoom = 10
 +
    maxZoom = 16
 +
    bbox = (-2, 50.0,1.0,52.0)
 +
    render_tiles(bbox, mapfile, tile_dir, minZoom, maxZoom)
 +
 
 +
    # Muenchen
 +
    bbox = (11.4,48.07, 11.7,48.22)
 +
    render_tiles(bbox, mapfile, tile_dir, 1, 12 , "Muenchen")
 +
 
 +
    # Muenchen+
 +
    bbox = (11.3,48.01, 12.15,48.44)
 +
    render_tiles(bbox, mapfile, tile_dir, 7, 12 , "Muenchen+")
 +
 
 +
    # Muenchen++
 +
    bbox = (10.92,47.7, 12.24,48.61)
 +
    render_tiles(bbox, mapfile, tile_dir, 7, 12 , "Muenchen++")
 +
 
 +
    # Nuernberg
 +
    bbox=(10.903198,49.560441,49.633534,11.038085)
 +
    render_tiles(bbox, mapfile, tile_dir, 10, 16, "Nuernberg")
 +
 
 +
    # Karlsruhe
 +
    bbox=(8.179113,48.933617,8.489252,49.081707)
 +
    render_tiles(bbox, mapfile, tile_dir, 10, 16, "Karlsruhe")
 +
 
 +
    # Karlsruhe+
 +
    bbox = (8.3,48.95,8.5,49.05)
 +
    render_tiles(bbox, mapfile, tile_dir, 1, 16, "Karlsruhe+")
 +
 
 +
    # Augsburg
 +
    bbox = (8.3,48.95,8.5,49.05)
 +
    render_tiles(bbox, mapfile, tile_dir, 1, 16, "Augsburg")
 +
 
 +
    # Augsburg+
 +
    bbox=(10.773251,48.369594,10.883834,48.438577)
 +
    render_tiles(bbox, mapfile, tile_dir, 10, 14, "Augsburg+")
 +
 
 +
    # Europe+
 +
    bbox = (1.0,10.0, 20.6,50.0)
 +
    render_tiles(bbox, mapfile, tile_dir, 1, 11 , "Europe+")
 +
</pre>
 +
 
 +
I select Munich++ first to see the result. Here is the invocation:
 +
 
 +
root@mapserver:/data/tiles# /usr/share/osm-mapnik/generate_tiles.py &
 +
[1] 11966
 +
root@mapserver:/data/tiles# render_tiles( (10.92, 47.7, 12.24, 48.61) /etc/mapnik-osm-data/osm.xml /var/www/tiles/osm/ 7 12 Muenchen++ )
 +
Muenchen++ : 8 135 89 
 +
Muenchen++ : 8 135 88 
 +
Muenchen++ : 7 68 44 
 +
Muenchen++ : 8 136 89 
 +
Muenchen++ : 9 271 176 
 +
Muenchen++ : 9 271 178 
 +
Muenchen++ : 9 271 177 
 +
Muenchen++ : 8 136 88 
 +
Muenchen++ : 9 272 176 
 +
Muenchen++ : 9 272 178 
 +
Muenchen++ : 9 273 176 
 +
Muenchen++ : 9 272 177 
 +
Muenchen++ : 10 543 353 
 +
Muenchen++ : 9 273 178 
 +
Muenchen++ : 9 273 177 
 +
Muenchen++ : 10 543 356 
 +
Muenchen++ : 10 543 355 
 +
Muenchen++ : 10 543 354 
 +
Muenchen++ : 10 543 357 
 +
Muenchen++ : 10 544 353 
 +
Muenchen++ : 10 544 356 
 +
Muenchen++ : 10 544 354 
 +
Muenchen++ : 10 544 355 
 +
Muenchen++ : 10 544 357 
 +
Muenchen++ : 10 545 353
 +
.......
 +
 
 +
Looks good. Still need to
 +
 
 +
chown -R www-data:www-data /var/www/tiles/osm/
 +
 
 +
Same, same, but different: I modify the generator like this in order to keep the multicore CPU busy:
 +
 
 +
    # Bayern++
 +
    bbox=(9.229,47.272,13.854,50.310)
 +
    render_tiles(bbox, mapfile, tile_dir, 8, 16, "Bayern++")
 +
 
 +
And starting it with
 +
 
 +
root@mapserver:~# nohup time /usr/share/osm-mapnik/generate_tiles.py &
 +
 
 +
Works fine.
 +
 
 +
== Create a OpenLayers Application ==
 +
 
 +
Create a file with this content at '''/var/www/map.html''':
 +
 
 +
<pre>
 +
<!DOCTYPE html>
 +
<html>
 +
  <head>
 +
    <link rel="stylesheet" href="ol.css" type="text/css">
 +
    <style>
 +
      .map { height: 95vh; width: 100%; }
 +
    </style>
 +
    <script src="ol.js" type="text/javascript"></script>
 +
    <title>OpenLayers example</title>
 +
  </head>
 +
  <body>
 +
    <div id="map" class="map"></div>
 +
    <script type="text/javascript">
 +
    var map = new ol.Map({
 +
      target: 'map',
 +
      layers: [
 +
        new ol.layer.Tile({
 +
            source: new ol.source.OSM({url: '/tiles/osm/{z}/{x}/{y}.png', maxZoom: 20})
 +
        })
 +
      ],
 +
      view: new ol.View({
 +
        center: ol.proj.fromLonLat([8, 53]),
 +
        zoom: 8
 +
      })
 +
    });
 +
    </script>
 +
  </body>
 +
</html>
 +
</pre>
  
−
=== OSM-Bright Configuration ===
+
I decided to download the CSS and JS files onto my document root so that a client will not need to access the internet for it:
  
 +
$ cd /var/www
 +
$ wget https://openlayers.org/en/v4.1.0/build/ol.js
 +
$ wget https://openlayers.org/en/v4.1.0/css/ol.css
 +
$ chown -R www-data:www-data *
  
−
=== Generate MBTiles with TileMill ===
+
=== Check it out! ===
  
 +
http://mapserver.db0hsr.ampr.org/map.html
  
−
=== Deploy MBTiles with TileStream ===
+
[[image:2017-12-31-mapserver1.png|600px]]

Aktuelle Version vom 9. Januar 2018, 04:38 Uhr

1 Scope

We have a really great high performance computer at DB0HSR, and since we're going to do several geography based experiments with it, we need our own mapserver to display positions.

There are other mapservers in the HAMNET:

2 Some basics

3 Mapserver Setup

I am replicating this report:

Other interesting guides:

3.1 Preparations

$ sudo apt-get install libmapnik-dev mapnik-utils git subversion \
dh-autoreconf apache2-dev apache2 unzip postgis make cmake g++ \
libboost-dev libboost-system-dev libboost-filesystem-dev \
libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libproj-dev \
lua5.2 liblua5.2-dev postgis imposm git postgresql postgresql-9.5-postgis-2.2 \
postgresql-9.5-postgis-scripts

3.2 Get and Prepare the Mapnik style

$ mkdir ~/src
$ cd ~/src

If behind a firewall, subversion will consider the file ~./subversion/servers. There check the settings: 

http-proxy-host = (your proxy IP address)
http-proxy-port = (your proxy portnumber)

Then check out the style: 

$ svn co http://svn.openstreetmap.org/applications/rendering/mapnik mapnik-style
$ cd ~/src/mapnik-style

If behind a firewall, wget will consider the environment variable http_proxy, in the format http://(proxy IP number):(proxy port number) 

$ sudo ./get-coastlines.sh /usr/local/share

will do something like this: 

root@mapserver:/home/dl8rds/src/mapnik-style# ./get-coastlines.sh /usr/local/share
--2017-12-29 03:50:41--  http://tile.openstreetmap.org/world_boundaries-spherical.tgz
Connecting to 44.225.41.33:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://planet.openstreetmap.org/historical-shapefiles/world_boundaries-spherical.tgz [following]
--2017-12-29 03:50:41--  http://planet.openstreetmap.org/historical-shapefiles/world_boundaries-spherical.tgz
Reusing existing connection to 44.225.41.33:8080.
Proxy request sent, awaiting response... 200 OK
Length: 52857349 (50M) [application/x-gzip]
Saving to: ‘/usr/local/share/world_boundaries-spherical.tgz’

/usr/local/share/world_boundaries-spherical.tg 100%[==================================================================================================>]  50,41M  1,12MB/s    in 44s     

2017-12-29 03:51:25 (1,14 MB/s) - ‘/usr/local/share/world_boundaries-spherical.tgz’ saved [52857349/52857349]

--2017-12-29 03:51:25--  http://tile.openstreetmap.org/processed_p.tar.bz2
Connecting to 44.225.41.33:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://planet.openstreetmap.org/historical-shapefiles/processed_p.tar.bz2 [following]
--2017-12-29 03:51:26--  http://planet.openstreetmap.org/historical-shapefiles/processed_p.tar.bz2
Reusing existing connection to 44.225.41.33:8080.
Proxy request sent, awaiting response... 200 OK
Length: 409468857 (390M) [application/x-bzip2]
Saving to: ‘/usr/local/share/processed_p.tar.bz2’

/usr/local/share/processed_p.tar.bz2           100%[==================================================================================================>] 390,50M  1,16MB/s    in 6m 21s  

2017-12-29 03:57:47 (1,03 MB/s) - ‘/usr/local/share/processed_p.tar.bz2’ saved [409468857/409468857]

--2017-12-29 03:57:47--  http://tile.openstreetmap.org/shoreline_300.tar.bz2
Connecting to 44.225.41.33:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://planet.openstreetmap.org/historical-shapefiles/shoreline_300.tar.bz2 [following]
--2017-12-29 03:57:47--  http://planet.openstreetmap.org/historical-shapefiles/shoreline_300.tar.bz2
Reusing existing connection to 44.225.41.33:8080.
Proxy request sent, awaiting response... 200 OK
Length: 43867136 (42M) [application/x-bzip2]
Saving to: ‘/usr/local/share/shoreline_300.tar.bz2’

/usr/local/share/shoreline_300.tar.bz2         100%[==================================================================================================>]  41,83M  1,29MB/s    in 34s     

2017-12-29 03:58:21 (1,23 MB/s) - ‘/usr/local/share/shoreline_300.tar.bz2’ saved [43867136/43867136]

--2017-12-29 03:58:21--  http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places.zip
Connecting to 44.225.41.33:8080... connected.
Proxy request sent, awaiting response... 302 Moved Temporarily
Location: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_populated_places.zip [following]
--2017-12-29 03:58:22--  http://naciscdn.org/naturalearth/10m/cultural/ne_10m_populated_places.zip
Reusing existing connection to 44.225.41.33:8080.
Proxy request sent, awaiting response... 200 OK
Length: 2002618 (1,9M) [application/x-zip-compressed]
Saving to: ‘/usr/local/share/ne_10m_populated_places.zip’

/usr/local/share/ne_10m_populated_places.zip   100%[==================================================================================================>]   1,91M   184KB/s    in 11s     

2017-12-29 03:58:34 (178 KB/s) - ‘/usr/local/share/ne_10m_populated_places.zip’ saved [2002618/2002618]

--2017-12-29 03:58:34--  http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip
Connecting to 44.225.41.33:8080... connected.
Proxy request sent, awaiting response... 302 Moved Temporarily
Location: http://naciscdn.org/naturalearth/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip [following]
--2017-12-29 03:58:35--  http://naciscdn.org/naturalearth/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip
Reusing existing connection to 44.225.41.33:8080.
Proxy request sent, awaiting response... 200 OK
Length: 45804 (45K) [application/x-zip-compressed]
Saving to: ‘/usr/local/share/ne_110m_admin_0_boundary_lines_land.zip’

/usr/local/share/ne_110m_admin_0_boundary_line 100%[==================================================================================================>]  44,73K  68,1KB/s    in 0,7s    

2017-12-29 03:58:35 (68,1 KB/s) - ‘/usr/local/share/ne_110m_admin_0_boundary_lines_land.zip’ saved [45804/45804]

world_boundaries/
world_boundaries/places.shx
world_boundaries/world_boundaries_m.index
world_boundaries/world_bnd_m.shx
world_boundaries/builtup_area.shx
world_boundaries/world_bnd_m.dbf
world_boundaries/builtup_area.prj
world_boundaries/places.shp
world_boundaries/world_boundaries_m.shx
world_boundaries/world_boundaries_m.shp
world_boundaries/places.dbf
world_boundaries/places.prj
world_boundaries/builtup_area.dbf
world_boundaries/world_bnd_m.shp
world_boundaries/world_bnd_m.prj
world_boundaries/world_boundaries_m.dbf
world_boundaries/builtup_area.shp
world_boundaries/world_boundaries_m.prj
world_boundaries/world_bnd_m.index
world_boundaries/builtup_area.index
processed_p.dbf
processed_p.index
processed_p.shp
processed_p.shx
shoreline_300.dbf
shoreline_300.index
shoreline_300.shp
shoreline_300.shx
Archive:  /usr/local/share/ne_10m_populated_places.zip
  inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.README.html  
 extracting: /usr/local/share/world_boundaries/ne_10m_populated_places.VERSION.txt  
 extracting: /usr/local/share/world_boundaries/ne_10m_populated_places.cpg  
  inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.dbf  
  inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.prj  
  inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.shp  
  inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.shx  
Archive:  /usr/local/share/ne_110m_admin_0_boundary_lines_land.zip
  inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.README.html  
 extracting: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.VERSION.txt  
 extracting: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.cpg  
  inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.dbf  
  inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.prj  
  inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.shp  
  inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.shx  
root@mapserver:/home/dl8rds/src/mapnik-style#

Now modify some mapnik definitions: 

$ cd inc
$ cp fontset-settings.xml.inc.template fontset-settings.xml.inc
$ cp datasource-settings.xml.inc.template datasource-settings.xml.inc
$ cp settings.xml.inc.template settings.xml.inc

Modify the file settings.xml.inc like this: 

<!--
Settings for symbols, the spatial reference of your postgis tables, coastline s$
-->

<!-- use 'symbols' unless you have moved the symbols directory -->
<!ENTITY symbols "symbols">

<!-- use the '&srs900913;' entity if you have called osm2pgsql without special $
<!ENTITY osm2pgsql_projection "&srs900913;">

<!-- used for 'node in way' ST_DWithin spatial operations -->
<!-- Use 0.1 (meters) when your database is in 900913     -->
<!-- Use 0.000001 (degrees) when your database is in 4326 -->
<!ENTITY dwithin_900913 "0.1">
<!ENTITY dwithin_4326 "0.00001">
<!ENTITY dwithin_node_way "&dwithin_900913;">

<!-- use 'world_boundaries', which is the usual naming for the local folder the$
<!ENTITY world_boundaries "/usr/local/share/world_boundaries">

<!-- use 'planet_osm' unless you have customized your database table prefix usi$
<!ENTITY prefix "planet_osm">

Modify the file datasource-settings.xml.inc like this: 

<!--
Settings for your postgres setup.

Note: feel free to leave password, host, port, or use blank
-->

<Parameter name="type">postgis</Parameter>
<!-- <Parameter name="password">%(password)s</Parameter> -->
<!-- <Parameter name="host"></Parameter> -->
<!-- <Parameter name="port">5432</Parameter> -->
<!-- <Parameter name="user">gis</Parameter> -->
<Parameter name="dbname">gis</Parameter>
<!-- this should be 'false' if you are manually providing the 'extent' -->
<Parameter name="estimate_extent">false</Parameter>
<!-- manually provided extent in epsg 900913 for whole globe -->
<!-- providing this speeds up Mapnik database queries -->
<Parameter name="extent">-20037508,-19929239,20037508,19929239</Parameter>

Finally copy the mapnik-style directory over to its default location at /etc/mapnik-osm-data: 

mv ~./src/mapnik-styles /etc/mapnik-osm-data

3.3 Install and Configure mod_tile and renderd

There are several recommendations to simply build and use mod_tile from source. I decided to generate a debian package first and then install the package properly. I followed this recommendation: 

https://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/

After you have installed mod_tile, there should be the following files: 

root@mapserver:/home/dl8rds/src# dpkg -L libapache2-mod-tile 
/.
/var
/var/www
/var/www/osm
/var/www/osm/slippymap.html
/usr
/usr/bin
/usr/bin/openstreetmap-tiles-update-expire
/usr/bin/osmosis-db_replag
/usr/share
/usr/share/munin
/usr/share/munin/plugins
/usr/share/munin/plugins/mod_tile_fresh
/usr/share/munin/plugins/mod_tile_latency
/usr/share/munin/plugins/mod_tile_zoom
/usr/share/munin/plugins/mod_tile_response
/usr/share/doc
/usr/share/doc/libapache2-mod-tile
/usr/share/doc/libapache2-mod-tile/copyright
/usr/share/doc/libapache2-mod-tile/changelog.Debian.gz
/usr/share/doc/libapache2-mod-tile/readme.txt.gz
/usr/lib
/usr/lib/apache2
/usr/lib/apache2/modules
/usr/lib/apache2/modules/mod_tile.so
/etc
/etc/apache2
/etc/apache2/mods-available
/etc/apache2/mods-available/tile.load
/etc/apache2/sites-available
/etc/apache2/sites-available/tileserver_site.conf

And the same with renderd: 

root@mapserver:/home/dl8rds/src# dpkg -L renderd
/.
/usr
/usr/bin
/usr/bin/render_old
/usr/bin/render_expired
/usr/bin/renderd
/usr/bin/render_speedtest
/usr/bin/render_list
/usr/share
/usr/share/munin
/usr/share/munin/plugins
/usr/share/munin/plugins/renderd_processed
/usr/share/munin/plugins/renderd_zoom_time
/usr/share/munin/plugins/renderd_zoom
/usr/share/munin/plugins/renderd_queue
/usr/share/munin/plugins/renderd_queue_time
/usr/share/doc
/usr/share/doc/renderd
/usr/share/doc/renderd/copyright
/usr/share/doc/renderd/changelog.Debian.gz
/usr/share/doc/renderd/readme.txt.gz
/usr/share/man
/usr/share/man/man8
/usr/share/man/man8/renderd.8.gz
/usr/share/man/man1
/usr/share/man/man1/render_expired.1.gz
/usr/share/man/man1/render_speedtest.1.gz
/usr/share/man/man1/render_old.1.gz
/usr/share/man/man1/render_list.1.gz
/usr/lib
/usr/lib/libiniparser.so.3.0.0
/usr/lib/libiniparser.la
/usr/lib/libiniparser.a
/etc
/etc/renderd.conf
/etc/init.d
/etc/init.d/renderd
/usr/lib/libiniparser.so.3
/usr/lib/libiniparser.so

Now edit the file /etc/renderd.conf like this: 

[renderd]
stats_file=/var/run/renderd/renderd.stats
socketname=/var/run/renderd/renderd.sock
num_threads=4
tile_dir=/var/www/tiles

[mapnik]
plugins_dir=/usr/lib/mapnik/3.0/input
font_dir=/usr/share/fonts
font_dir_recurse=true

[osm]
HOST=localhost
TILESIZE=256
URI=/tiles/osm/
XML=/etc/mapnik-osm-data/osm.xml
DESCRIPTION=This is the standard osm mapnik style
HOST=mapserver.db0hsr.apr.org

Finally ensure that the modules are used by Apache: 

$ sudo sh -c 'echo "LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so" > /etc/apache2/mods-available/tile.load'

Activate it without an Apache restart: 

$ sudo a2enmod tile

Now modify the website definition. Decide what to do with the default website config 000-default.conf. I simply deleted it. 

$ sudo nano /etc/apache2/sites-available/tileserver_site.conf

Make sure that the file looks like this: 

<VirtualHost *:80>
    ServerAdmin dl8rds@darc.de
    ServerName mapserver.db0hsr.ampr.org
    DocumentRoot /var/www
    LogLevel info
    AddTileConfig /tiles/ osm
    LoadTileConfigFile /etc/renderd.conf
    ModTileRequestTimeout 3
    ModTileMissingRequestTimeout 10
    ModTileMaxLoadOld 2
    ModTileMaxLoadMissing 5
    ModTileRenderdSocketName /var/run/renderd/renderd.sock
    ModTileCacheDurationMax 604800
    ModTileCacheDurationDirty 900
    ModTileCacheDurationMinimum 10800
    ModTileCacheDurationMediumZoom 13 86400
    ModTileCacheDurationLowZoom 9 518400
    ModTileCacheLastModifiedFactor 0.20
    ModTileEnableTileThrottling Off
    ModTileEnableTileThrottlingXForward 0
    ModTileThrottlingTiles 10000 1 
    ModTileThrottlingRenders 128 0.2
        <Directory />
		Options FollowSymLinks
		AllowOverride None
	</Directory>
	<Directory /var/www/>
		Options Indexes FollowSymLinks MultiViews
		AllowOverride None
		Order allow,deny
		allow from all
	</Directory>
	ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
	<Directory "/usr/lib/cgi-bin">
		AllowOverride None
		Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
		Order allow,deny
		Allow from all
	</Directory>
    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>
</VirtualHost>

Finally restart Apache: 

$ sudo apachectl -t

and it should say something like this: 

Syntax OK

Oh yes, don't forget to create the runtime directory for renderd:

$ sudo mkdir /var/run/renderd /var/lib/mod_tile $ sudo chown www-data:www-data /var/run/renderd /var/lib/mod_tile

3.4 Download Map Data

Get the dataset for Germany. A company called Geofabrik is providing these files: 

http://download.geofabrik.de/europe/germany.html

Now download:

$ mkdir ~/osm && cd ~/osm wget http://download.geofabrik.de/europe/germany-latest.osm.pbf

3.5 Prepare the Database

There are packages for Ubuntu. You can compile the program yourself, too. I prefer the prefabricated package. 

$ sudo apt-get install osm2pgsql

Then prepare the database: 

$ sudo -u postgres createuser gis
$ sudo -u postgres createdb -E UTF8 -O gis gis
$ sudo -u postgres createlang plpgsql gis

Edit the file /etc/postgresql/9.5/main/pg_hba.conf: 

# "local" is for Unix domain socket connections only
local   all         all                               trust
local   gis         gis                               trust

There is a script that also installs postgis: 

/usr/share/doc/osm2pgsql/examples/install-postgis-osm-db.sh

Unfortunately it does not work correctly. So let's do it by hand: 

$ sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'

Correct table ownerships: 

$ echo 'ALTER TABLE geometry_columns OWNER TO gis; ALTER TABLE spatial_ref_sys OWNER TO gis;' | sudo -u postgres psql gis

And then import the database: 

$ osm2pgsql --create --database gis --username gis --slim --cache 4096 --hstore /data/germany-latest.osm.pbf

Now wait for a couple of days.

3.6 Start everything

Make sure you have the output directory hierarchy: 

$ mkdir /var/www/tiles/osm
$ chown -R www-data:www-data /var/www/tiles/osm

Simply make sure all the services are up and running: 

$ sudo service renderd start
$ sudo systemctl restart apache2

3.7 Precompute Tiles

First of all, let me state that I have simply decided to base all my maps on Mapnik in order to keep the prcess comparable and simple. It works, and never change a winning team. Not for now.

3.7.1 Method 1: Tirex with Mapnik -- Abandoned.

This could be done with Tirex, but I still have some problems with it. So let's check render_list...

3.7.2 Method 2: render_list with Mapnik

This is my render_list invocation: 

root@mapserver:~# render_list -v -n 6 -all --socket=/var/run/renderd/renderd.sock --min-zoom=6 --max-zoom=15 -x 2112 -X 2208 -y 2624 -Y 2880

... but I'm still playing around with it. Our machine is powerful, but it's not a supercomputer. :-) Here is some good advice (in German): 

https://forum.openstreetmap.org/viewtopic.php?id=20800

3.7.3 Method 3: generate_tiles.py with Mapnik

Here is the decription how to do that: https://sites.google.com/site/nitinpasumarthy/blog/createyourowntileserverandmapclient

This is the region I'm interested in: http://www.openstreetmap.org/export#map=7/48.756/12.398&layers=C

Here is my python file: /usr/share/osm-mapnik/generate_tiles.py

And I see that it can be modified via environment variables, notably:

  • MAPNIK_MAP_FILE, which is /etc/mapnik-osm-data/osm.xml
  • MAPNIK_TILE_DIR, which is /var/www/tiles/osm

and this section of the file is also interesting: 

    bbox = (-180.0,-90.0, 180.0,90.0)

    render_tiles(bbox, mapfile, tile_dir, 0, 5, "World")

    minZoom = 10
    maxZoom = 16
    bbox = (-2, 50.0,1.0,52.0)
    render_tiles(bbox, mapfile, tile_dir, minZoom, maxZoom)

    # Muenchen
    bbox = (11.4,48.07, 11.7,48.22)
    render_tiles(bbox, mapfile, tile_dir, 1, 12 , "Muenchen")

    # Muenchen+
    bbox = (11.3,48.01, 12.15,48.44)
    render_tiles(bbox, mapfile, tile_dir, 7, 12 , "Muenchen+")

    # Muenchen++
    bbox = (10.92,47.7, 12.24,48.61)
    render_tiles(bbox, mapfile, tile_dir, 7, 12 , "Muenchen++")

    # Nuernberg
    bbox=(10.903198,49.560441,49.633534,11.038085)
    render_tiles(bbox, mapfile, tile_dir, 10, 16, "Nuernberg")

    # Karlsruhe
    bbox=(8.179113,48.933617,8.489252,49.081707)
    render_tiles(bbox, mapfile, tile_dir, 10, 16, "Karlsruhe")

    # Karlsruhe+
    bbox = (8.3,48.95,8.5,49.05)
    render_tiles(bbox, mapfile, tile_dir, 1, 16, "Karlsruhe+")

    # Augsburg
    bbox = (8.3,48.95,8.5,49.05)
    render_tiles(bbox, mapfile, tile_dir, 1, 16, "Augsburg")

    # Augsburg+
    bbox=(10.773251,48.369594,10.883834,48.438577)
    render_tiles(bbox, mapfile, tile_dir, 10, 14, "Augsburg+")

    # Europe+
    bbox = (1.0,10.0, 20.6,50.0)
    render_tiles(bbox, mapfile, tile_dir, 1, 11 , "Europe+")

I select Munich++ first to see the result. Here is the invocation: 

root@mapserver:/data/tiles# /usr/share/osm-mapnik/generate_tiles.py &
[1] 11966
root@mapserver:/data/tiles# render_tiles( (10.92, 47.7, 12.24, 48.61) /etc/mapnik-osm-data/osm.xml /var/www/tiles/osm/ 7 12 Muenchen++ )
Muenchen++ : 8 135 89  
Muenchen++ : 8 135 88  
Muenchen++ : 7 68 44  
Muenchen++ : 8 136 89  
Muenchen++ : 9 271 176  
Muenchen++ : 9 271 178  
Muenchen++ : 9 271 177  
Muenchen++ : 8 136 88  
Muenchen++ : 9 272 176  
Muenchen++ : 9 272 178  
Muenchen++ : 9 273 176  
Muenchen++ : 9 272 177  
Muenchen++ : 10 543 353  
Muenchen++ : 9 273 178  
Muenchen++ : 9 273 177  
Muenchen++ : 10 543 356  
Muenchen++ : 10 543 355  
Muenchen++ : 10 543 354  
Muenchen++ : 10 543 357  
Muenchen++ : 10 544 353  
Muenchen++ : 10 544 356  
Muenchen++ : 10 544 354  
Muenchen++ : 10 544 355  
Muenchen++ : 10 544 357  
Muenchen++ : 10 545 353
.......

Looks good. Still need to 

chown -R www-data:www-data /var/www/tiles/osm/

Same, same, but different: I modify the generator like this in order to keep the multicore CPU busy: 

   # Bayern++
   bbox=(9.229,47.272,13.854,50.310)
   render_tiles(bbox, mapfile, tile_dir, 8, 16, "Bayern++")

And starting it with 

root@mapserver:~# nohup time /usr/share/osm-mapnik/generate_tiles.py &

Works fine.

4 Create a OpenLayers Application

Create a file with this content at /var/www/map.html: 

<!DOCTYPE html>
<html>
  <head>
    <link rel="stylesheet" href="ol.css" type="text/css">
    <style>
      .map { height: 95vh; width: 100%; }
    </style>
    <script src="ol.js" type="text/javascript"></script>
    <title>OpenLayers example</title>
  </head>
  <body>
    <div id="map" class="map"></div>
    <script type="text/javascript">
    var map = new ol.Map({
      target: 'map',
      layers: [
        new ol.layer.Tile({
            source: new ol.source.OSM({url: '/tiles/osm/{z}/{x}/{y}.png', maxZoom: 20})
        })
      ],
      view: new ol.View({
        center: ol.proj.fromLonLat([8, 53]),
        zoom: 8
      })
    });
    </script>
  </body>
</html>

I decided to download the CSS and JS files onto my document root so that a client will not need to access the internet for it: 

$ cd /var/www
$ wget https://openlayers.org/en/v4.1.0/build/ol.js
$ wget https://openlayers.org/en/v4.1.0/css/ol.css
$ chown -R www-data:www-data *

4.1 Check it out!

http://mapserver.db0hsr.ampr.org/map.html

2017-12-31-mapserver1.png

Abgerufen von „http://www.dl8rds.de/index.php?title=My_Own_Mapserver&oldid=2894“