Search:
Login
Preferences
Help/Guide
About Trac
Register
Forgot your password?
Wiki
Timeline
Changelog
Browse Source
View Tickets
New Ticket
Roadmap
Builds
Sonar
Search
Context Navigation
+1
Start Page
Index
History
Editing SharedTileCache
Adjust edit area height:
8
12
16
20
24
28
32
36
40
Edit side-by-side
= Shared Tile Cache = This document describes a common caching layout that can be used to share imagery tiles among different applications. Status:: Draft Author/s:: Yuri D'Elia Last revised:: 2014-03-23 === Minor revisions === Lists of concerns addressed since the initial draft: - 2014-03-24: Name of the cache "ID"/directory restricted to 20 characters (Robert Norris, from Viking development) - 2014-03-24: Spaces removed from Windows/OSX shared tile root (Don-vip, JOSM) - 2014-03-25: Use an "OSM" prefix for the shared tile root to limit namespace squatting (Kevin Krammer, Marble) To encourage discussions about the spec, please don't edit this page directly. Use a relevant mailing list or this bug report: #9813 == Premise == We have several desktop applications that are querying various tile servers and cache their imagery locally, however each application is currently storing and managing their cache independently. This leads to a huge waste of space (especially for aerial imagery) in the form of duplicated caches with essentially the same content, and bandwidth due to needless server re-querying. In this document we describe a common cache format, layout and methods to access, create and modify tile caches that applications may choose to adhere to share their imagery cache. == Cache directory root == === Unix/Linux === The cache directory root is located in the {{{$XDG_CACHE_HOME/osm/tiles}}} (usually "~/.cache/osm/tiles"). See the [http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html XDG Base Directory Specification] to locate this directory correctly. === Windows XP === {{{%UserProfile%\Local Settings\Application Data\OSM\SharedTileCache\}}} === Windows Vista, 7, 8 === {{{%UserProfile%\AppData\Local\OSM\SharedTileCache\}}} === Mac OS X === {{{NSCachesDirectory/OSM/SharedTileCache}}} (usually "~/Library/Caches/OSM/SharedTileCache"). Use the NSPathUtilities to locate the cache directory using [https://developer.apple.com/library/mac/documentation/Cocoa/Reference/Foundation/Miscellaneous/Foundation_Constants/Reference/reference.html NSCachesDirectory]. == Tile directory layout == Under the cache directory root, each directory contains an independent tile cache directory. Each directory can be any valid arbitrary directory name, but ''should'' be an understandable short ID for the tile provider (such as "Mapnik", "Bing", etc). Applications '''must not''' rely on the short ID name, but '''must''' instead parse the cache properties as outlined in [#CacheDiscovery]. The content of the tile cache is the same as followed by [osmwiki:Slippy_map_tilenames Slippy Map], with the difference that each file can be located by the pattern "/zoom/x/y.extension". Each tile can ''optionally'' have additional metadata associated with it, contained in the file following the pattern "/zoom/x/y.extension.ini", as discussed in [#TileManagement]. At the root level, each cache '''must''' provide a [#CacheProperties] file, named "/cache.ini". This file provides several know attributes, such as the "extension" required to locate each tile. An example cache directory layout is outlined below: {{{ ~/.cache/osm/tiles/ ~/.cache/osm/tiles/Mapnik/ ~/.cache/osm/tiles/Mapnik/cache.ini ~/.cache/osm/tiles/Mapnik/14/8709/5792.png ~/.cache/osm/tiles/Mapnik/14/8709/5793.png.ini ~/.cache/osm/tiles/Bing/ ~/.cache/osm/tiles/Bing/cache.ini ... }}} == Cache properties == #CacheProperties The "Cache properties" file defines data associated with the cache itself. The file is named "cache.ini" an is located at the root level of the cache: {{{~/.cache/osm/tiles/Mapnik/cache.ini}}} The file format is a simple text file, UTF-8 encoded, with each line containing a "key=value" pair. Applications '''must''' ignore any key that they don't know. Applications '''must''' also preserve all keys when the file is being updated. Any other arbitrary key can be included. The following '''mandatory''' keys are defined: name:: A long descriptive name of the tile provider for display purposes. url:: Base URL of the tile provider. type:: URL schema of the tile provider (currently only "TMS") extension:: Image file extension ("png" or "jpg"). size:: Maximal size of the cache in bytes, with 0 meaning unlimited, and -1 meaning no appending. age:: Minimum tile age (in seconds) An example properties file is shown below: {{{ name=OSM Mapnik url=http://tile.openstreetmap.org type=TMS extension=png size=0 age=604800 }}} == Cache discovery == #CacheDiscovery When initializing cache management, applications willing to use the shared tile cache should list all available caches inside the shared tile cache, and parse their respective properties files. Applications '''must not''' depend on the ID (given by the directory name), but should instead locate the cache by ''matching'' on "url", "type" and possibly "extension". If a suitable match is found, the ''existing'' ID/directory should be used for further access. If no suitable directory can be found, a ''new'' directory can be safely created inside the shared root. The application is free to choose any valid directory name, but ''should'' use a short, explicative name which '''must''' be less or equal to 20 characters. == Tile management == #TileManagement Assuming an empty tile cache, the application is free to create any file by downloading it from the provider and using the appropriate "/zoom/x/y.extension" pattern, creating missing directories. Additional data can be stored for each tile, using the file named "/zoom/x/y.extension.ini". This file follows the same convention as the [#CacheProperties] file, and is a simple UTF-8 encoded "key=value" file containing arbitrary keys which are application-defined. Again, applications '''must''' ignore unknown keys, ''can'' add new keys, but '''must''' also preserve any existing one when updating the content. Applications are free to create/update the metadata file at any time. When probing the cache for an existing tile, the application '''must''' check for the presence of the tile image "/zoom/x/y.extension" and check it's modification time. If the tile has been created ''or'' modified less than the specified cache age, the application '''must''' use the existing tile. If the tile is older, it ''can'' be considered stale and the application is free to query the server again, supplying an "If-Modified-Since" header. For this reason, applications '''must not''' modify the file modification time of the tile image except when updating the tile. When an existing tile is being ''updated or removed'', any associated "/zoom/x/y.extension.ini" file '''must''' be deleted as well, irregardless of the contents. When an existing tile is being deleted, in addition to removing the associated "ini" file, the application must also try to remove the "/zoom/x/" and "/zoom/" directories when empty. When the cache size is defined as 0, applications '''must not''' perform any cache pruning. If size is anything greater than 0, each application can choose ''his own'' policy and schedule for tile expiration, deleting each tile in turn until the size of the ''entire directory tree'' is smaller or equal to the defined cache size. When the cache size is defined as -1, applications '''must not''' include any additional content into the cache, but are free to query it. == Tile metadata == Tile metadata can be used for any purpose where storing additional information for each tile can be useful. As such, the following keys are defined to be known: etag:: ETag as returned by the server (used for querying servers that don't support If-Modified-Since). Tile metadata can also be useful to store information about tile usage (view count) to improve usage-based cache pruning, though no keys are standardized for this behavior yet. == Concurrent access == To enable "opportunistic" concurrent cache access to each tile cache, we define the following simple rules: * After deciding to create or update a tile, the image file ("/zoom/x/y.extension") '''must''' be moved ''atomically'' into the directory tree (and not created in-place), ''replacing'' any existing file (if any). An attempt can then be made to read the associated "ini" file. The "ini" file '''must be ignored''' if it's create/modification time is earlier than the image file and '''must''' then be deleted if not replaced. * When deleting an existing tile, the image file must be removed first, followed by an attempt to remove the "ini" file, and the x/zoom directories in turn. * Applications '''must not''' expect written tiles (or their metadata) to exist on disk or be the same as previously written. * Updates to the cache properties and tile metadata '''must''' be similarly performed using atomic replacement. Since no locking is performed, applications '''must not''' cache their contents. With this approach, a tile being updated concurrently may remove existing metadata, however doesn't require any locking or sophisticated metadata syncronization. Moreover, it doesn't break the assumption that metadata is optional while guaranteeing that any data (when present) is not stale.
Note:
See
WikiFormatting
and
TracWiki
for help on editing wiki content.
Change information
Your email or username:
E-mail address and name can be saved in the
Preferences
Comment about this change (optional):
Note:
See
TracWiki
for help on using the wiki.