Docker imports the planet is really slow

[myrainbowandsky]

system: Ubuntu 20.2
RAM 512GB
disk: 2TB
cores: 24cores.

How to rocket the planet importation speed?

I am using docker to import planet like this:

docker volume create openstreetmap-data
time docker run -v /home/renderaccount/zambia-latest.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/12/main overv/openstreetmap-tile-server:1.3.10 import

But the importation speed is really slow like this:

+ '[' 1 -ne 1 ']'
 + '[' import = import ']'
 + chown postgres:postgres -R /var/lib/postgresql
 + '[' '!' -f /var/lib/postgresql/12/main/PG_VERSION ']'
 + createPostgresConfig
 + cp /etc/postgresql/12/main/postgresql.custom.conf.tmpl /etc/postgresql/12/main/conf.d/postgresql.custom.conf
 + sudo -u postgres echo 'autovacuum = on'
 + cat /etc/postgresql/12/main/conf.d/postgresql.custom.conf
 # Suggested minimal settings from
 # https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/
 
 shared_buffers = 128MB
 min_wal_size = 1GB
 max_wal_size = 2GB
 maintenance_work_mem = 256MB
 
 # Suggested settings from
 # https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45
 
 max_connections = 250
 temp_buffers = 32MB
 work_mem = 128MB
 wal_buffers = 1024kB
 wal_writer_delay = 500ms
 commit_delay = 10000
 # checkpoint_segments = 60 # unrecognized in psql 10.7.1
 max_wal_size = 2880MB
 random_page_cost = 1.1
 track_activity_query_size = 16384
 autovacuum_vacuum_scale_factor = 0.05
 autovacuum_analyze_scale_factor = 0.02
 
 listen_addresses = '*'
 autovacuum = on
 + service postgresql start
  * Starting PostgreSQL 12 database server                               [ OK ] 
 + sudo -u postgres createuser renderer
 createuser: error: creation of new role failed: ERROR:  role "renderer" already exists
 + sudo -u postgres createdb -E UTF8 -O renderer gis
 createdb: error: database creation failed: ERROR:  database "gis" already exists
 + sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
 ERROR:  extension "postgis" already exists
 + sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
 ERROR:  extension "hstore" already exists
 + sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
 ALTER TABLE
 + sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
 ALTER TABLE
 + setPostgresPassword
 + sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '\''renderer'\'''
 ALTER ROLE
 + '[' '!' -f /data.osm.pbf ']'
 + '[' disabled = enabled ']'
 + '[' -f /data.poly ']'
 + sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua --number-processes 4 -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf
 osm2pgsql version 1.2.0 (64 bit id space)
 
 Allocating memory for dense node cache
 Allocating dense node cache in one big chunk
 Allocating memory for sparse node cache
 Sharing dense sparse
 Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=11
 Mid: pgsql, cache=800
 Setting up table: planet_osm_nodes
 Setting up table: planet_osm_ways
 Setting up table: planet_osm_rels
 Using lua based tag processing pipeline with script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua
 Using projection SRS 3857 (Spherical Mercator)
 Setting up table: planet_osm_point
 Setting up table: planet_osm_line
 Setting up table: planet_osm_polygon
 Setting up table: planet_osm_roads
 
 Reading in file: /data.osm.pbf
 Using PBF parser.
Processing: Node((base) wentao@white:~$ sudo docker logs 07feb9c51fc8 | grep pattern
(base) wentao@white:~$ sudo docker logs 07feb9c51fc8 | grep pattern
(base) wentao@white:~$ sudo docker logs 07feb9c51fc8 -f
+ '[' 1 -ne 1 ']'
+ '[' import = import ']'
+ chown postgres:postgres -R /var/lib/postgresql
+ '[' '!' -f /var/lib/postgresql/12/main/PG_VERSION ']'
+ createPostgresConfig
+ cp /etc/postgresql/12/main/postgresql.custom.conf.tmpl /etc/postgresql/12/main/conf.d/postgresql.custom.conf
+ sudo -u postgres echo 'autovacuum = on'
+ cat /etc/postgresql/12/main/conf.d/postgresql.custom.conf
# Suggested minimal settings from
# https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/

shared_buffers = 128MB
min_wal_size = 1GB
max_wal_size = 2GB
maintenance_work_mem = 256MB

# Suggested settings from
# https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45

max_connections = 250
temp_buffers = 32MB
work_mem = 128MB
wal_buffers = 1024kB
wal_writer_delay = 500ms
commit_delay = 10000
# checkpoint_segments = 60 # unrecognized in psql 10.7.1
max_wal_size = 2880MB
random_page_cost = 1.1
track_activity_query_size = 16384
autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02

listen_addresses = '*'
autovacuum = on
+ service postgresql start
 * Starting PostgreSQL 12 database server                                [ OK ] 
+ sudo -u postgres createuser renderer
createuser: error: creation of new role failed: ERROR:  role "renderer" already exists
+ sudo -u postgres createdb -E UTF8 -O renderer gis
createdb: error: database creation failed: ERROR:  database "gis" already exists
+ sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
ERROR:  extension "postgis" already exists
+ sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
ERROR:  extension "hstore" already exists
+ sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
ALTER TABLE
+ sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
ALTER TABLE
+ setPostgresPassword
+ sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '\''renderer'\'''
ALTER ROLE
+ '[' '!' -f /data.osm.pbf ']'
+ '[' disabled = enabled ']'
+ '[' -f /data.poly ']'
+ sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua --number-processes 4 -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf
osm2pgsql version 1.2.0 (64 bit id space)

Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=11
Mid: pgsql, cache=800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using lua based tag processing pipeline with script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads

Reading in file: /data.osm.pbf
Using PBF parser.
Processing: Node(7417817k 157.1k/s) Way(19416k 0.05k/s) Relation(0 0.00/s)

[pnorman]

Few questions

• How long does it take to do the processing stage?
• Does this happen with more recent osm2pgsql versions?
• What happens when you change the --cache you are giving osm2pgsql?
• What types of disks do you have? Disk capacity does not matter unless you run out of room.

[lonvia]

Duplicate of Overv/openstreetmap-tile-server#329 and better discussed there.