Better deal with large shapefiles

Created on Tuesday 17 November 2020, 07:45

Back to task list
  • ID
    511352
  • Project
    Metabolism of Cities Data Hub
  • Status
    Open
  • Priority
    Medium
  • Type
    Programming work
  • Assigned to
    No one yet
  • Subscribers
    Paul Hoekman

You are not logged in

Log In Register

Please join us and let's build things, together!

Description

We're now getting many more shapefiles loaded into the system, which is great. However, some are 10... 20.... 30... 50 MB files! This becomes crazy large to load on a single map. Also, pulling in all that geometry takes time in the back-end, so in general we must just better figure out what to do with this. Also, I've seen some crazy detailed files about very specific places (the type of paint used throughout the port of Barcelona, stuff like that). So we must also question what we need and what we don't need. The reference spaces table is growing by leaps and bounds which again is great but 100k items just to describe this single port seems excessive. So let's figure out how we best approach this.

Discussion and updates

Paul Hoekman Paul Hoekman Johannesburg
4 years ago

New task was created

Paul Hoekman Paul Hoekman Johannesburg
4 years ago

Task was assigned to Paul Hoekman

Paul Hoekman Paul Hoekman Johannesburg
4 years ago

Status change: Open → In Progress

Paul Hoekman Paul Hoekman Johannesburg
4 years ago

Some first changes include:



  • When a user wants to process a huge shapefile (> 1000 items), a warning pops up, see attached

  • When a map is opened that is already processed but that contains > 500 items, then the system will only show the first 500 items on the map

  • When a map is opened that is more than 5 / 10 / 20 MB, then the system will use a simplification script that makes lines in the polygon less detailed but much smaller (the level of simplification increases depending on the file size). The user is still given an option to open the non-simplified map, but it can take ages so the user is warned.

Paul Hoekman Paul Hoekman Johannesburg
1 year, 1 month ago

I've stepped down from my Metabolism of Cities involvement but if another volunteer wants to pick this up later and ask me any question about this, feel free to reach out and I'll try to assist where I can.

Paul Hoekman Paul Hoekman Johannesburg
1 year, 1 month ago

Paul Hoekman is no longer responsible for this task