{"action":"created","issue":{"url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/801","repository_url":"https://api.github.com/repos/openstreetmap/osm2pgsql","labels_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/801/labels{/name}","comments_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/801/comments","events_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/801/events","html_url":"https://github.com/openstreetmap/osm2pgsql/issues/801","id":275111023,"number":801,"title":"Using LuaJIT to speed up lua code","user":{"login":"mmd-osm","id":5842757,"avatar_url":"https://avatars1.githubusercontent.com/u/5842757?v=4","gravatar_id":"","url":"https://api.github.com/users/mmd-osm","html_url":"https://github.com/mmd-osm","followers_url":"https://api.github.com/users/mmd-osm/followers","following_url":"https://api.github.com/users/mmd-osm/following{/other_user}","gists_url":"https://api.github.com/users/mmd-osm/gists{/gist_id}","starred_url":"https://api.github.com/users/mmd-osm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mmd-osm/subscriptions","organizations_url":"https://api.github.com/users/mmd-osm/orgs","repos_url":"https://api.github.com/users/mmd-osm/repos","events_url":"https://api.github.com/users/mmd-osm/events{/privacy}","received_events_url":"https://api.github.com/users/mmd-osm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-11-18T21:24:22Z","updated_at":"2018-02-16T19:53:26Z","closed_at":null,"author_association":"NONE","body":"LuaJIT is a powerful drop in replacement to run Lua code in native mode with JIT optimizations: see http://luajit.org/luajit.html for details. To evaluate the potential impact on osm2pgsql runtimes, I did some very quick & dirty prototyping. Primary goal was to have a working osm2pgsql , and being able to do some initial run time comparison between the current and a LuaJIT enabled version.\r\n\r\nTest was using the latest openstreetmap-carto style on a very small country extract (30MB PBF). Platform: Core i7 and Ubuntu 16.04, 64bit.\r\n\r\n`osm2pgsql -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis  extract.osm.pbf`\r\n\r\nBefore (Lua)\r\n`Processing: Node(3045k 1015.2k/s) Way(556k 46.37k/s) Relation(1790 1790.00/s)  parse time: 16s`\r\n\r\nAfter (LuaJIT)\r\n`Processing: Node(3045k 3045.5k/s) Way(556k 139.12k/s) Relation(3410 3410.00/s)  parse time: 6s`\r\n\r\nFirst conclusion: LuaJIT allows for some significant savings in the osm2pgsql parsing phase. I'd recommend to further evaluate this option and see how it might be integrated into osm2pgsql.\r\n\r\n## Code\r\n\r\nAdd'ntl cmake file I used (couldn't find an official one yet): https://github.com/ranisalt/forgottenserver/blob/luajit-optional/cmake/FindLuaJIT.cmake\r\n\r\n```C++\r\ndiff --git a/CMakeLists.txt b/CMakeLists.txt\r\nindex 0b62652..b837ba3 100644\r\n--- a/CMakeLists.txt\r\n+++ b/CMakeLists.txt\r\n@@ -11,7 +11,8 @@ list(APPEND CMAKE_MODULE_PATH \"${CMAKE_CURRENT_SOURCE_DIR}/cmake\")\r\n set(DATA_DIR \\\"${CMAKE_INSTALL_PREFIX}/share/osm2pgsql\\\")\r\n \r\n option(BUILD_TESTS \"Build test suite\" OFF)\r\n-option(WITH_LUA    \"Build with lua support\" ON)\r\n+option(WITH_LUA    \"Build with Lua support\" ON)\r\n+option(WITH_LUAJIT \"Build with LuaJIT support\" OFF)\r\n \r\n if (NOT TESTING_TIMEOUT)\r\n   set(TESTING_TIMEOUT 1200)\r\n@@ -85,7 +86,12 @@ find_package(Osmium REQUIRED COMPONENTS io proj)\r\n include_directories(SYSTEM ${OSMIUM_INCLUDE_DIRS})\r\n \r\n if (WITH_LUA)\r\n-  find_package(Lua REQUIRED)\r\n+  if (WITH_LUAJIT)\r\n+    find_package(LuaJIT REQUIRED)\r\n+    set(HAVE_LUAJIT 1)\r\n+  else()\r\n+    find_package(Lua REQUIRED)\r\n+  endif()\r\n   include_directories(${LUA_INCLUDE_DIR})\r\n   set(HAVE_LUA 1)\r\n endif()\r\n@@ -108,7 +114,7 @@ find_package(Threads)\r\n \r\n set (LIBS ${Boost_LIBRARIES} ${PostgreSQL_LIBRARY} ${OSMIUM_LIBRARIES})\r\n \r\n-if (LUA_FOUND)\r\n+if (LUA_FOUND OR LUAJIT_FOUND)\r\n   list(APPEND LIBS ${LUA_LIBRARIES})\r\n endif()\r\n \r\n@@ -212,7 +218,7 @@ set(osm2pgsql_lib_SOURCES\r\n   wkb.hpp\r\n )\r\n \r\n-if (LUA_FOUND)\r\n+if (LUA_FOUND OR LUAJIT_FOUND)\r\n   list(APPEND osm2pgsql_lib_SOURCES tagtransform-lua.cpp)\r\n endif()\r\n \r\n```\r\n\r\n\r\n\r\nNB: I do realize that OSRM removed LuaJIT support last year (see: Rip out LuaJIT for good #2955), due to lack of usage in their (Mapbox) own  setup, unclear impact wrt speed improvement and users reporting crashes. At least to some extent this seems to be related to the their build system, and automatic fallback to LuaJIT. That's why I'd propose `WITH_LUAJIT` should be disabled by default."},"comment":{"url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/comments/366341304","html_url":"https://github.com/openstreetmap/osm2pgsql/issues/801#issuecomment-366341304","issue_url":"https://api.github.com/repos/openstreetmap/osm2pgsql/issues/801","id":366341304,"user":{"login":"StyXman","id":167327,"avatar_url":"https://avatars0.githubusercontent.com/u/167327?v=4","gravatar_id":"","url":"https://api.github.com/users/StyXman","html_url":"https://github.com/StyXman","followers_url":"https://api.github.com/users/StyXman/followers","following_url":"https://api.github.com/users/StyXman/following{/other_user}","gists_url":"https://api.github.com/users/StyXman/gists{/gist_id}","starred_url":"https://api.github.com/users/StyXman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StyXman/subscriptions","organizations_url":"https://api.github.com/users/StyXman/orgs","repos_url":"https://api.github.com/users/StyXman/repos","events_url":"https://api.github.com/users/StyXman/events{/privacy}","received_events_url":"https://api.github.com/users/StyXman/received_events","type":"User","site_admin":false},"created_at":"2018-02-16T19:53:26Z","updated_at":"2018-02-16T19:53:26Z","author_association":"CONTRIBUTOR","body":"I have to say, I had the same thing. I imported twice the whole Europe using this lua style: https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/style.lua . I also used the null output driver, as I don't have mush free space to do it for real. \r\n\r\nAs @sav123 , I didn't notice any improvements, so probably @mmd-osm can tell us where or what to look? I definitely like the idea of the PR, speeding up imports, but if it actually has no (noticeable?) impact, I'm not sure it's worth it.\r\n\r\n@mmd-osm just a question: did you try first without luajit and the with it? maybe the speed difference has more to do with your source file being already cached or something?"}}
{"actor":{"display_login":"StyXman"}}