My Own Mapserver: Unterschied zwischen den Versionen
Dl8rds (Diskussion | Beiträge) (→Precompute Tiles) |
Dl8rds (Diskussion | Beiträge) (→Create a OpenLayers Application) |
||
Zeile 501: | Zeile 501: | ||
Here is the decription how to do that: https://sites.google.com/site/nitinpasumarthy/blog/createyourowntileserverandmapclient | Here is the decription how to do that: https://sites.google.com/site/nitinpasumarthy/blog/createyourowntileserverandmapclient | ||
− | + | == Create a OpenLayers Application == | |
Create a file with this content at '''/var/www/map.html''': | Create a file with this content at '''/var/www/map.html''': |
Version vom 9. Januar 2018, 04:05 Uhr
Inhaltsverzeichnis
- 1 Scope
- 2 Some basics
- 3 Mapserver Setup
- 3.1 Preparations
- 3.2 Get and Prepare the Mapnik style
- 3.3 Install and Configure mod_tile and renderd
- 3.4 Download Map Data
- 3.5 Prepare the Database
- 3.6 Start everything
- 3.7 Precompute Tiles
- 3.8 Method 1: Tirex with Mapnik -- Abandoned.
- 3.9 Method 2: render_list with Mapnik
- 3.10 Method 3: generate_tiles.py with Mapnik
- 4 Create a OpenLayers Application
1 Scope
We have a really great high performance computer at DB0HSR, and since we're going to do several geography based experiments with it, we need our own mapserver to display positions.
There are other mapservers in the HAMNET:
2 Some basics
3 Mapserver Setup
I am replicating this report:
Other interesting guides:
3.1 Preparations
$ sudo apt-get install libmapnik-dev mapnik-utils git subversion \ dh-autoreconf apache2-dev apache2 unzip postgis make cmake g++ \ libboost-dev libboost-system-dev libboost-filesystem-dev \ libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libproj-dev \ lua5.2 liblua5.2-dev postgis imposm git postgresql postgresql-9.5-postgis-2.2 \ postgresql-9.5-postgis-scripts
3.2 Get and Prepare the Mapnik style
$ mkdir ~/src $ cd ~/src
If behind a firewall, subversion will consider the file ~./subversion/servers. There check the settings:
http-proxy-host = (your proxy IP address) http-proxy-port = (your proxy portnumber)
Then check out the style:
$ svn co http://svn.openstreetmap.org/applications/rendering/mapnik mapnik-style $ cd ~/src/mapnik-style
If behind a firewall, wget will consider the environment variable http_proxy, in the format http://(proxy IP number):(proxy port number)
$ sudo ./get-coastlines.sh /usr/local/share
will do something like this:
root@mapserver:/home/dl8rds/src/mapnik-style# ./get-coastlines.sh /usr/local/share --2017-12-29 03:50:41-- http://tile.openstreetmap.org/world_boundaries-spherical.tgz Connecting to 44.225.41.33:8080... connected. Proxy request sent, awaiting response... 301 Moved Permanently Location: http://planet.openstreetmap.org/historical-shapefiles/world_boundaries-spherical.tgz [following] --2017-12-29 03:50:41-- http://planet.openstreetmap.org/historical-shapefiles/world_boundaries-spherical.tgz Reusing existing connection to 44.225.41.33:8080. Proxy request sent, awaiting response... 200 OK Length: 52857349 (50M) [application/x-gzip] Saving to: ‘/usr/local/share/world_boundaries-spherical.tgz’ /usr/local/share/world_boundaries-spherical.tg 100%[==================================================================================================>] 50,41M 1,12MB/s in 44s 2017-12-29 03:51:25 (1,14 MB/s) - ‘/usr/local/share/world_boundaries-spherical.tgz’ saved [52857349/52857349] --2017-12-29 03:51:25-- http://tile.openstreetmap.org/processed_p.tar.bz2 Connecting to 44.225.41.33:8080... connected. Proxy request sent, awaiting response... 301 Moved Permanently Location: http://planet.openstreetmap.org/historical-shapefiles/processed_p.tar.bz2 [following] --2017-12-29 03:51:26-- http://planet.openstreetmap.org/historical-shapefiles/processed_p.tar.bz2 Reusing existing connection to 44.225.41.33:8080. Proxy request sent, awaiting response... 200 OK Length: 409468857 (390M) [application/x-bzip2] Saving to: ‘/usr/local/share/processed_p.tar.bz2’ /usr/local/share/processed_p.tar.bz2 100%[==================================================================================================>] 390,50M 1,16MB/s in 6m 21s 2017-12-29 03:57:47 (1,03 MB/s) - ‘/usr/local/share/processed_p.tar.bz2’ saved [409468857/409468857] --2017-12-29 03:57:47-- http://tile.openstreetmap.org/shoreline_300.tar.bz2 Connecting to 44.225.41.33:8080... connected. Proxy request sent, awaiting response... 301 Moved Permanently Location: http://planet.openstreetmap.org/historical-shapefiles/shoreline_300.tar.bz2 [following] --2017-12-29 03:57:47-- http://planet.openstreetmap.org/historical-shapefiles/shoreline_300.tar.bz2 Reusing existing connection to 44.225.41.33:8080. Proxy request sent, awaiting response... 200 OK Length: 43867136 (42M) [application/x-bzip2] Saving to: ‘/usr/local/share/shoreline_300.tar.bz2’ /usr/local/share/shoreline_300.tar.bz2 100%[==================================================================================================>] 41,83M 1,29MB/s in 34s 2017-12-29 03:58:21 (1,23 MB/s) - ‘/usr/local/share/shoreline_300.tar.bz2’ saved [43867136/43867136] --2017-12-29 03:58:21-- http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places.zip Connecting to 44.225.41.33:8080... connected. Proxy request sent, awaiting response... 302 Moved Temporarily Location: http://naciscdn.org/naturalearth/10m/cultural/ne_10m_populated_places.zip [following] --2017-12-29 03:58:22-- http://naciscdn.org/naturalearth/10m/cultural/ne_10m_populated_places.zip Reusing existing connection to 44.225.41.33:8080. Proxy request sent, awaiting response... 200 OK Length: 2002618 (1,9M) [application/x-zip-compressed] Saving to: ‘/usr/local/share/ne_10m_populated_places.zip’ /usr/local/share/ne_10m_populated_places.zip 100%[==================================================================================================>] 1,91M 184KB/s in 11s 2017-12-29 03:58:34 (178 KB/s) - ‘/usr/local/share/ne_10m_populated_places.zip’ saved [2002618/2002618] --2017-12-29 03:58:34-- http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip Connecting to 44.225.41.33:8080... connected. Proxy request sent, awaiting response... 302 Moved Temporarily Location: http://naciscdn.org/naturalearth/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip [following] --2017-12-29 03:58:35-- http://naciscdn.org/naturalearth/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip Reusing existing connection to 44.225.41.33:8080. Proxy request sent, awaiting response... 200 OK Length: 45804 (45K) [application/x-zip-compressed] Saving to: ‘/usr/local/share/ne_110m_admin_0_boundary_lines_land.zip’ /usr/local/share/ne_110m_admin_0_boundary_line 100%[==================================================================================================>] 44,73K 68,1KB/s in 0,7s 2017-12-29 03:58:35 (68,1 KB/s) - ‘/usr/local/share/ne_110m_admin_0_boundary_lines_land.zip’ saved [45804/45804] world_boundaries/ world_boundaries/places.shx world_boundaries/world_boundaries_m.index world_boundaries/world_bnd_m.shx world_boundaries/builtup_area.shx world_boundaries/world_bnd_m.dbf world_boundaries/builtup_area.prj world_boundaries/places.shp world_boundaries/world_boundaries_m.shx world_boundaries/world_boundaries_m.shp world_boundaries/places.dbf world_boundaries/places.prj world_boundaries/builtup_area.dbf world_boundaries/world_bnd_m.shp world_boundaries/world_bnd_m.prj world_boundaries/world_boundaries_m.dbf world_boundaries/builtup_area.shp world_boundaries/world_boundaries_m.prj world_boundaries/world_bnd_m.index world_boundaries/builtup_area.index processed_p.dbf processed_p.index processed_p.shp processed_p.shx shoreline_300.dbf shoreline_300.index shoreline_300.shp shoreline_300.shx Archive: /usr/local/share/ne_10m_populated_places.zip inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.README.html extracting: /usr/local/share/world_boundaries/ne_10m_populated_places.VERSION.txt extracting: /usr/local/share/world_boundaries/ne_10m_populated_places.cpg inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.dbf inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.prj inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.shp inflating: /usr/local/share/world_boundaries/ne_10m_populated_places.shx Archive: /usr/local/share/ne_110m_admin_0_boundary_lines_land.zip inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.README.html extracting: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.VERSION.txt extracting: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.cpg inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.dbf inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.prj inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.shp inflating: /usr/local/share/world_boundaries/ne_110m_admin_0_boundary_lines_land.shx root@mapserver:/home/dl8rds/src/mapnik-style#
Now modify some mapnik definitions:
$ cd inc $ cp fontset-settings.xml.inc.template fontset-settings.xml.inc $ cp datasource-settings.xml.inc.template datasource-settings.xml.inc $ cp settings.xml.inc.template settings.xml.inc
Modify the file settings.xml.inc like this:
<!-- Settings for symbols, the spatial reference of your postgis tables, coastline s$ --> <!-- use 'symbols' unless you have moved the symbols directory --> <!ENTITY symbols "symbols"> <!-- use the '&srs900913;' entity if you have called osm2pgsql without special $ <!ENTITY osm2pgsql_projection "&srs900913;"> <!-- used for 'node in way' ST_DWithin spatial operations --> <!-- Use 0.1 (meters) when your database is in 900913 --> <!-- Use 0.000001 (degrees) when your database is in 4326 --> <!ENTITY dwithin_900913 "0.1"> <!ENTITY dwithin_4326 "0.00001"> <!ENTITY dwithin_node_way "&dwithin_900913;"> <!-- use 'world_boundaries', which is the usual naming for the local folder the$ <!ENTITY world_boundaries "/usr/local/share/world_boundaries"> <!-- use 'planet_osm' unless you have customized your database table prefix usi$ <!ENTITY prefix "planet_osm">
Modify the file datasource-settings.xml.inc like this:
<!-- Settings for your postgres setup. Note: feel free to leave password, host, port, or use blank --> <Parameter name="type">postgis</Parameter> <!-- <Parameter name="password">%(password)s</Parameter> --> <!-- <Parameter name="host"></Parameter> --> <!-- <Parameter name="port">5432</Parameter> --> <!-- <Parameter name="user">gis</Parameter> --> <Parameter name="dbname">gis</Parameter> <!-- this should be 'false' if you are manually providing the 'extent' --> <Parameter name="estimate_extent">false</Parameter> <!-- manually provided extent in epsg 900913 for whole globe --> <!-- providing this speeds up Mapnik database queries --> <Parameter name="extent">-20037508,-19929239,20037508,19929239</Parameter>
Finally copy the mapnik-style directory over to its default location at /etc/mapnik-osm-data:
mv ~./src/mapnik-styles /etc/mapnik-osm-data
3.3 Install and Configure mod_tile and renderd
There are several recommendations to simply build and use mod_tile from source. I decided to generate a debian package first and then install the package properly. I followed this recommendation:
https://switch2osm.org/serving-tiles/building-a-tile-server-from-packages/
After you have installed mod_tile, there should be the following files:
root@mapserver:/home/dl8rds/src# dpkg -L libapache2-mod-tile /. /var /var/www /var/www/osm /var/www/osm/slippymap.html /usr /usr/bin /usr/bin/openstreetmap-tiles-update-expire /usr/bin/osmosis-db_replag /usr/share /usr/share/munin /usr/share/munin/plugins /usr/share/munin/plugins/mod_tile_fresh /usr/share/munin/plugins/mod_tile_latency /usr/share/munin/plugins/mod_tile_zoom /usr/share/munin/plugins/mod_tile_response /usr/share/doc /usr/share/doc/libapache2-mod-tile /usr/share/doc/libapache2-mod-tile/copyright /usr/share/doc/libapache2-mod-tile/changelog.Debian.gz /usr/share/doc/libapache2-mod-tile/readme.txt.gz /usr/lib /usr/lib/apache2 /usr/lib/apache2/modules /usr/lib/apache2/modules/mod_tile.so /etc /etc/apache2 /etc/apache2/mods-available /etc/apache2/mods-available/tile.load /etc/apache2/sites-available /etc/apache2/sites-available/tileserver_site.conf
And the same with renderd:
root@mapserver:/home/dl8rds/src# dpkg -L renderd /. /usr /usr/bin /usr/bin/render_old /usr/bin/render_expired /usr/bin/renderd /usr/bin/render_speedtest /usr/bin/render_list /usr/share /usr/share/munin /usr/share/munin/plugins /usr/share/munin/plugins/renderd_processed /usr/share/munin/plugins/renderd_zoom_time /usr/share/munin/plugins/renderd_zoom /usr/share/munin/plugins/renderd_queue /usr/share/munin/plugins/renderd_queue_time /usr/share/doc /usr/share/doc/renderd /usr/share/doc/renderd/copyright /usr/share/doc/renderd/changelog.Debian.gz /usr/share/doc/renderd/readme.txt.gz /usr/share/man /usr/share/man/man8 /usr/share/man/man8/renderd.8.gz /usr/share/man/man1 /usr/share/man/man1/render_expired.1.gz /usr/share/man/man1/render_speedtest.1.gz /usr/share/man/man1/render_old.1.gz /usr/share/man/man1/render_list.1.gz /usr/lib /usr/lib/libiniparser.so.3.0.0 /usr/lib/libiniparser.la /usr/lib/libiniparser.a /etc /etc/renderd.conf /etc/init.d /etc/init.d/renderd /usr/lib/libiniparser.so.3 /usr/lib/libiniparser.so
Now edit the file /etc/renderd.conf like this:
[renderd] stats_file=/var/run/renderd/renderd.stats socketname=/var/run/renderd/renderd.sock num_threads=4 tile_dir=/var/www/tiles [mapnik] plugins_dir=/usr/lib/mapnik/3.0/input font_dir=/usr/share/fonts font_dir_recurse=true [osm] HOST=localhost TILESIZE=256 URI=/tiles/osm/ XML=/etc/mapnik-osm-data/osm.xml DESCRIPTION=This is the standard osm mapnik style HOST=mapserver.db0hsr.apr.org
Finally ensure that the modules are used by Apache:
$ sudo sh -c 'echo "LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so" > /etc/apache2/mods-available/tile.load'
Activate it without an Apache restart:
$ sudo a2enmod tile
Now modify the website definition. Decide what to do with the default website config 000-default.conf. I simply deleted it.
$ sudo nano /etc/apache2/sites-available/tileserver_site.conf
Make sure that the file looks like this:
<VirtualHost *:80> ServerAdmin dl8rds@darc.de ServerName mapserver.db0hsr.ampr.org DocumentRoot /var/www LogLevel info AddTileConfig /tiles/ osm LoadTileConfigFile /etc/renderd.conf ModTileRequestTimeout 3 ModTileMissingRequestTimeout 10 ModTileMaxLoadOld 2 ModTileMaxLoadMissing 5 ModTileRenderdSocketName /var/run/renderd/renderd.sock ModTileCacheDurationMax 604800 ModTileCacheDurationDirty 900 ModTileCacheDurationMinimum 10800 ModTileCacheDurationMediumZoom 13 86400 ModTileCacheDurationLowZoom 9 518400 ModTileCacheLastModifiedFactor 0.20 ModTileEnableTileThrottling Off ModTileEnableTileThrottlingXForward 0 ModTileThrottlingTiles 10000 1 ModTileThrottlingRenders 128 0.2 <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory /var/www/> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all </Directory> ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ <Directory "/usr/lib/cgi-bin"> AllowOverride None Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch Order allow,deny Allow from all </Directory> Alias /doc/ "/usr/share/doc/" <Directory "/usr/share/doc/"> Options Indexes MultiViews FollowSymLinks AllowOverride None Order deny,allow Deny from all Allow from 127.0.0.0/255.0.0.0 ::1/128 </Directory> </VirtualHost>
Finally restart Apache:
$ sudo apachectl -t
and it should say something like this:
Syntax OK
Oh yes, don't forget to create the runtime directory for renderd:
$ sudo mkdir /var/run/renderd /var/lib/mod_tile $ sudo chown www-data:www-data /var/run/renderd /var/lib/mod_tile
3.4 Download Map Data
Get the dataset for Germany. A company called Geofabrik is providing these files:
http://download.geofabrik.de/europe/germany.html
Now download:
$ mkdir ~/osm && cd ~/osm wget http://download.geofabrik.de/europe/germany-latest.osm.pbf
3.5 Prepare the Database
There are packages for Ubuntu. You can compile the program yourself, too. I prefer the prefabricated package.
$ sudo apt-get install osm2pgsql
Then prepare the database:
$ sudo -u postgres createuser gis $ sudo -u postgres createdb -E UTF8 -O gis gis $ sudo -u postgres createlang plpgsql gis
Edit the file /etc/postgresql/9.5/main/pg_hba.conf:
# "local" is for Unix domain socket connections only local all all trust local gis gis trust
There is a script that also installs postgis:
/usr/share/doc/osm2pgsql/examples/install-postgis-osm-db.sh
Unfortunately it does not work correctly. So let's do it by hand:
$ sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'
Correct table ownerships:
$ echo 'ALTER TABLE geometry_columns OWNER TO gis; ALTER TABLE spatial_ref_sys OWNER TO gis;' | sudo -u postgres psql gis
And then import the database:
$ osm2pgsql --create --database gis --username gis --slim --cache 4096 --hstore /data/germany-latest.osm.pbf
Now wait for a couple of days.
3.6 Start everything
Make sure you have the output directory hierarchy:
$ mkdir /var/www/tiles/osm $ chown -R www-data:www-data /var/www/tiles/osm
Simply make sure all the services are up and running:
$ sudo service renderd start $ sudo systemctl restart apache2
3.7 Precompute Tiles
First of all, let me state that I have simply decided to base all my maps on Mapnik in order to keep the prcess comparable and simple. It works, and never change a winning team. Not for now.
3.8 Method 1: Tirex with Mapnik -- Abandoned.
This could be done with Tirex, but I still have some problems with it. So let's check render_list...
3.9 Method 2: render_list with Mapnik
This is my render_list invocation:
root@mapserver:~# render_list -v -n 6 -all --socket=/var/run/renderd/renderd.sock --min-zoom=6 --max-zoom=15 -x 2112 -X 2208 -y 2624 -Y 2880
... but I'm still playing around with it. Our machine is powerful, but it's not a supercomputer. :-) Here is some good advice (in German):
https://forum.openstreetmap.org/viewtopic.php?id=20800
3.10 Method 3: generate_tiles.py with Mapnik
Here is the decription how to do that: https://sites.google.com/site/nitinpasumarthy/blog/createyourowntileserverandmapclient
4 Create a OpenLayers Application
Create a file with this content at /var/www/map.html:
<!DOCTYPE html> <html> <head> <link rel="stylesheet" href="ol.css" type="text/css"> <style> .map { height: 95vh; width: 100%; } </style> <script src="ol.js" type="text/javascript"></script> <title>OpenLayers example</title> </head> <body> <div id="map" class="map"></div> <script type="text/javascript"> var map = new ol.Map({ target: 'map', layers: [ new ol.layer.Tile({ source: new ol.source.OSM({url: '/tiles/osm/{z}/{x}/{y}.png', maxZoom: 20}) }) ], view: new ol.View({ center: ol.proj.fromLonLat([8, 53]), zoom: 8 }) }); </script> </body> </html>
I decided to download the CSS and JS files onto my document root so that a client will not need to access the internet for it:
$ cd /var/www $ wget https://openlayers.org/en/v4.1.0/build/ol.js $ wget https://openlayers.org/en/v4.1.0/css/ol.css $ chown -R www-data:www-data *