{"ok": true, "next": null, "next_url": null, "rows": [{"path": "/tutorials/clean-data", "title": "Cleaning data with sqlite-utils and Datasette", "body": "Tutorials > Cleaning data with sqlite-utils and Datasette\n\nCleaning data with sqlite-utils and Datasette\n\nThis tutorial will show you how to use the sqlite-utils command line tool import data into SQLite, clean it and then explore it with Datasette.\n\nThe concepts in this tutorial are also demonstrated in this video of a talk I gave at the HYTRADBOI conference in April 2022.\n\n\n\nStart with some CSV data\nThe Florida Fish and Wildlife Conservation Commission maintain a database of manatee carcass recovery locations, with details dating back to April 1974.\nYou can request the latest copy as a CSV file from their website - or download this copy of the CSV: Manatee_Carcass_Recovery_Locations_in_Florida.csv generated on July 31st 2022.\nInstall sqlite-utils\nHere are the installation instructions. If you are on a Mac with Homebrew installed you can run brew install sqlite-utils. Python users can run pip install sqlite-utils.\nImport the CSV into a SQLite database\nsqlite-utils is a command line tool, so you'll need to open up a terminal on your computer and change directory to the place where you saved the CSV file.\nYou can create a new manatees.db SQLite database like this:\nsqlite-utils insert manatees.db locations \\\n  Manatee_Carcass_Recovery_Locations_in_Florida.csv --csv -d\n\nThis sqlite-utils insert command inserts data into a database table, creating the table if necessary.\n\nmanatees.db is the SQLite database file we are using - this will be created if it doesn't exist yet.\nlocations is the name of the table we are creating\nManatee_Carcass_Recovery_Locations_in_Florida.csv is the CSV file we are importing\n--csv tells the tool to treat the incoming data as CSV data\nThe -d option tells the tool to automatically detect the types of data. Without this, every column will be treated as text. With this option columns that have numeric data in will be used to populate numeric columns.\n\nHaving created the database, we can use the schema command to view its schema:\nsqlite-utils schema manatees.db\n\nThe output of that command looks like this:\nCREATE TABLE \"locations\" (\n   [X] FLOAT,\n   [Y] FLOAT,\n   [OBJECTID] INTEGER,\n   [FIELDID] TEXT,\n   [REPDATE] TEXT,\n   [REPYEAR] INTEGER,\n   [REPMONTH] INTEGER,\n   [REPDAY] INTEGER,\n   [SEX] TEXT,\n   [TLENGTH] FLOAT,\n   [STATE] TEXT,\n   [COUNTY] TEXT,\n   [LAT] FLOAT,\n   [LONG_] FLOAT,\n   [DCODE] INTEGER,\n   [MORTALITY] TEXT,\n   [created_user] TEXT,\n   [created_date] TEXT,\n   [last_edited_user] TEXT,\n   [last_edited_date] TEXT\n);\n\nThe locations table here has been created with columns that match the columns that were present in the CSV file. The columns have the appropriate types thanks to the -d option we passed to sqlite-utils insert.\nWe can use the sqlite-utils tables command to see those tables:\nsqlite-utils tables manatees.db --counts\n\nOutput is:\n[{\"table\": \"locations\", \"count\": 13568}]\n\nThe --counts option adds a count of the rows in each table. That's over 13,000 deceased manatees!\nBy default, the tables command outputs JSON. You can change this to be a table format using the --table or -t options:\nsqlite-utils tables manatees.db --counts -t\ntable        count\n---------  -------\nlocations    13568\n\nQuerying the data\nLet's run a SQL query against your data. The sqlite-utils command to do that looks like this:\nsqlite-utils manatees.db \"select * from locations limit 2\"\n\nThis outputs JSON, showing the first two rows and every column (thanks to that select *):\n[{\"X\": -80.1176098026034, \"Y\": 26.1004059191504, \"OBJECTID\": 149433, \"FIELDID\": \"M7401\", \"REPDATE\": \"1974/04/03 00:00:00+00\", \"REPYEAR\": 1974, \"REPMONTH\": 4, \"REPDAY\": 3, \"SEX\": \"F\", \"TLENGTH\": 260.0, \"STATE\": \"FL\", \"COUNTY\": \"Broward\", \"LAT\": 26.100401, \"LONG_\": -80.117607, \"DCODE\": 9, \"MORTALITY\": \"Undetermined: Other\", \"created_user\": \"FWC\", \"created_date\": \"2021/10/15 11:30:55+00\", \"last_edited_user\": \"FWC\", \"last_edited_date\": \"2021/10/15 11:30:55+00\"},\n {\"X\": -80.1664354041335, \"Y\": 25.8337161801256, \"OBJECTID\": 149434, \"FIELDID\": \"M7402\", \"REPDATE\": \"1974/06/27 00:00:00+00\", \"REPYEAR\": 1974, \"REPMONTH\": 6, \"REPDAY\": 27, \"SEX\": \"M\", \"TLENGTH\": 290.0, \"STATE\": \"FL\", \"COUNTY\": \"Miami-Dade\", \"LAT\": 25.833711, \"LONG_\": -80.166433, \"DCODE\": 1, \"MORTALITY\": \"Human Related: Watercraft Collision\", \"created_user\": \"FWC\", \"created_date\": \"2021/10/15 11:30:55+00\", \"last_edited_user\": \"FWC\", \"last_edited_date\": \"2021/10/15 11:30:55+00\"}]\n\nLet's specify some columns, limit to 10 and add -t to output the results as a table.\nsqlite-utils manatees.db -t \\\n  \"select REPDATE, MORTALITY, COUNTY from locations limit 10\"\n\nThis outputs:\nREPDATE                 MORTALITY                            COUNTY\n----------------------  -----------------------------------  ------------\n1974/04/03 00:00:00+00  Undetermined: Other                  Broward\n1974/06/27 00:00:00+00  Human Related: Watercraft Collision  Miami-Dade\n1974/08/20 00:00:00+00  Human Related: Watercraft Collision  Miami-Dade\n1974/10/23 00:00:00+00  Human Related: Other                 Volusia\n1974/11/11 00:00:00+00  Human Related: Other                 Brevard\n1974/12/25 00:00:00+00  Human Related: Watercraft Collision  Citrus\n1974/12/27 00:00:00+00  Undetermined: Too Decomposed         Indian River\n1975/01/01 00:00:00+00  Verified: Not Necropsied             Brevard\n1975/01/19 00:00:00+00  Human Related: Watercraft Collision  Miami-Dade\n1975/01/31 00:00:00+00  Verified: Not Necropsied             Brevard\n\nOpening it in Datasette\nDatasette provides a browser-based interface for exploring a SQLite database. Install that, (brew install datasette works for Homebrew users) and run it against your database:\ndatasette manatees.db\n\nThen navigate to http://localhost:8001/ to start exploring your data.\nTransforming the columns\nExploring the data helps identify improvements we can make to the way it is structured.\nThe LAT and LONG_ column look like they might be latitudes and longitudes. Renaming those columns to latitude and longitude will make that more obvious, and will also let us visualize the data using a Datasette plugin in a moment.\nThe created_user, last_edited_user and STATE columns aren't interesting: they always store the same value.\nThe X and Y columns are duplicates of the latitude and longitude.\nFinally, the FIELDID column looks to be a unique identifier for every row in the database. This would make a good primary key for our table.\nThe sqlite-utils transform command can be used to apply transformations to a table - rename columns, dropping columns, assigning primary keys and more.\nThe following command will make a bunch of changes to the table structure in one go:\nsqlite-utils transform manatees.db locations \\\n  --rename LAT latitude \\\n  --rename LONG_ longitude \\\n  --drop created_user \\\n  --drop last_edited_user \\\n  --drop X \\\n  --drop Y \\\n  --drop STATE \\\n  --drop OBJECTID \\\n  --pk FIELDID\n\nNow run sqlite-utils schema manatees.db to see the result of those changes:\nCREATE TABLE \"locations\" (\n   [FIELDID] TEXT PRIMARY KEY,\n   [REPDATE] TEXT,\n   [REPYEAR] INTEGER,\n   [REPMONTH] INTEGER,\n   [REPDAY] INTEGER,\n   [SEX] TEXT,\n   [TLENGTH] FLOAT,\n   [COUNTY] TEXT,\n   [latitude] FLOAT,\n   [longitude] FLOAT,\n   [DCODE] INTEGER,\n   [MORTALITY] TEXT,\n   [created_date] TEXT,\n   [last_edited_date] TEXT\n);\n\nVisualizing the locations on a map\nDatasette supports plugins, which enable extra features such as data visualizations.\nThe datasette-cluster-map plugin runs against any table with a latitude and longitude column, displaying them as points on a map. This is why we renamed those LAT and LONG_ columns earlier!\nTo install the plugin, quit the Datasette server (using Ctrl+C in the terminal window) and run the following:\ndatasette install datasette-cluster-map\n\nThen start Datasette running again:\ndatasette manatees.db\n\nIf you visit http://localhost:8001/manatees/locations you will now see the first 1,000 rows as points on the map:\n\nClick the \"load all\" button to load and display all 13,000+ locations.\nFixing the date columns\nThe date columns currently look like this:\n2021/10/15 11:30:55+00\n\nThis is using YYYY/MM/DD format (at least it's not that confusing - for Europeans - American MM/DD/YYYY thing). I prefer the ISO standard format YYYY-MM-DD - so let's convert it.\nThe sqlite-utils transform command can be used to apply a transformation function to the values in one or more columns.\nHere's how to use that to parse miscellaneous date and time formats and turn them into that standard ISO format:\nsqlite-utils convert manatees.db locations \\\n  REPDATE created_date last_edited_date \\\n  'r.parsedatetime(value)'\n\nThis runs against the REPDATE, created_date and last_edited_date columns. It will use the built-in r.parsedatetime() recipe (documented here) to convert those values and save them back to the same column.\nRun that command, then view the data in Datasette again. Those columns have now been converted!\nExtracting columns into a separate table\nThere's one more improvement we will make to our data.\nOpen the table in Datasette, then use either the suggested facets list or the cog menu at the top of the columns to facet by both DCODE and MORTALITY.\n\nThe result counts shown for the two facets are the same. This suggests that these two columns represent the same data - a DCODE of 1 corresponds to a MORTALITY of \"Human Related: Watercraft Collision\", 2 means \"Human Related: Flood Gate/Canal Lock\" and so on.\nWe can use the sqlite-utils extract command to extract these columns into a separate table.\nsqlite-utils extract manatees.db locations DCODE MORTALITY \\\n  --rename MORTALITY name \\\n  --table mortality\n\nRunning this will create a new table called mortality and populate it with one row for each of the DCODE and MORTALITY pairs in the database. It removes those columns from the locations table and replaces them with a mortality_id foreign key column pointing to the new table.\nHere's the output of sqlite-utils schema manatees.db showing the impact of those changes on the schema:\nCREATE TABLE [mortality] (\n  [id] INTEGER PRIMARY KEY,\n  [DCODE] INTEGER,\n  [name] TEXT\n);\nCREATE TABLE \"locations\" (\n  [FIELDID] TEXT PRIMARY KEY,\n  [REPDATE] TEXT,\n  [REPYEAR] INTEGER,\n  [REPMONTH] INTEGER,\n  [REPDAY] INTEGER,\n  [SEX] TEXT,\n  [TLENGTH] FLOAT,\n  [COUNTY] TEXT,\n  [latitude] FLOAT,\n  [longitude] FLOAT,\n  [mortality_id] INTEGER,\n  [created_date] TEXT,\n  [last_edited_date] TEXT,\n  FOREIGN KEY([mortality_id]) REFERENCES [mortality]([id])\n);\nCREATE UNIQUE INDEX [idx_mortality_DCODE_name]\n   ON [mortality] ([DCODE], [name]);\n\nFurther reading\nsqlite-utils offers a lot more tools for cleaning up data. Check out this documentation and these blog posts for more details.\n\nsqlite-utils command line tool documentation\nExecuting advanced ALTER TABLE operations in SQLite\nApply conversion functions to data in SQLite columns with the sqlite-utils CLI tool\nRefactoring databases with sqlite-utils extract"}, {"path": "/tutorials/codespaces", "title": "Using Datasette in GitHub Codespaces", "body": "Tutorials > Using Datasette in GitHub Codespaces\n\nUsing Datasette in GitHub Codespaces\nGitHub Codespaces is a free tool from GitHub that gives you a full Linux development environment running in the cloud, accessed entirely through your browser.\nIt's a really neat way to get started with Datasette, in particular as a tool to build your own private Datasette instance and start using it to explore and analyze data.\nPlease note that Codespaces is not designed to host public websites. Any Datasette instances you create within a Codespace will only be accessible as private instances for your personal use.\nThis tutorial will show how to install and run Datasette in a Codespaces environment.\nStep 1: Create a Codespace\nEach Codespace needs to be attached to a GitHub repository.\nYou can use any of your existing repositories, or create a brand new one.\nOr if you don't want to create your own repository you can use my simonw/codespaces-datasette repository instead.\nUse the \"Code\" button, select the \"Codespaces\" tab and then click \"Create codespace on main\" to start a new codespace.\n\nAfter a short delay while the codespace starts running you should see an interface that looks like this:\n\nWe will be working exclusively in the Terminal panel at the bottom of the page.\nStep 2: Install Datasette\nBefore we can run Datasette, we need to install it. We can do that using the following command:\npipx install datasette\n\nThis will install Datasette using pipx, which ensures it is kept in its own separate environment where it can't clash with any other Python code you may decide to run in your codespace.\nTo confirm that Datasette has installed successfully, run this:\ndatasette --version\n\nWe also need to install a plugin:\ndatasette install datasette-codespaces\n\nThe datasette-codespaces plugin makes some small changes to Datasette to better run in the Codespaces environment. You can run Datasette without it but you may run into some frustrating issues, such as internal links not working correctly.\nStep 3: Start Datasette\nTo start Datasette, run this command:\ndatasette data.db --create\n\nThis will start Datasette running against an empty SQLite database called data.db. If that database does not exist yet it will be created.\nYou should see the following:\n\nClick that \"Open in Browser\" button to open the Datasette web interface in a new browser tab.\nIf you can't see that button, check to see if the \"Ports\" tab is available in the Codespaces interface. You can use that to connect to your Datasette instance instead.\nStep 4: Import some data\nYour Datasette interface will start with an empty database. Let's add some data to it!\nWe'll do that using the sqlite-utils command line tool.\nWe can leave Datasette running in the background by starting a new terminal. Click on the word \"python\" at the top of the terminal window and select \"Split terminal\":\n\nYou should now have two terminal windows next to each other.\nIn the new terminal window, run this command to install sqlite-utils:\npipx install sqlite-utils\n\nThen confirm installation with:\nsqlite-utils --version\n\n\nLet's fetch a CSV file from the web:\nwget https://static.simonwillison.net/static/2022/Manatee_Carcass_Recovery_Locations_in_Florida.csv\nAnd insert it into our database:\nsqlite-utils insert data.db locations \\\n    Manatee_Carcass_Recovery_Locations_in_Florida.csv \\\n    --csv -d\n    This creates a new table called locations containing the data from that CSV file - manatee carcass recovery locations in Florida dating back to 1974.\nThe --csv option tells sqlite-utils to treat the file as a CSV file, and the -d option tells it to infer the column types from the data (rather than treating every column as a text column).\nSee the tutorial Cleaning data with sqlite-utils and Datasette for more about this example data.\nStep 5: Install some more plugins\nDatasette has over 100 available plugins, listed in the plugins directory.\nTo install these, we first need to stop the Datasette server running. To do this first click back on the terminal window running Datasette and then hit Ctrl+C to stop the server.\nThe datasette install ... command installs new plugins. Let's install a few of them:\ndatasette install \\\n  datasette-vega \\\n  datasette-cluster-map \\\n  datasette-copyable \\\n  datasette-configure-fts \\\n  datasette-edit-schema \\\n  datasette-upload-csvs\n\nHere's what each of these plugins does:\n\ndatasette-vega adds line, bar and scatter chart visualizations\ndatasette-cluster-map adds a map visualization for any table with latitude and longitude columns\ndatasette-copyable provides additional export options for copying and pasting data out of Datasette\ndatasette-configure-fts helps you to configure full-text search for your tables\ndatasette-edit-schema provides tools toedit the schema of your tables\n\nStart the server running again like this:\ndatasette data.db\nWe don't need the --create option any more as we know the database already exists - although it won't break anything if we do include it.\nStep 6: Explore those manatee locations on a map\nIn Datasette, navigate to that \"locations\" table. It should have columns called X and Y which look like they might contain latitude and longitude data.\nThe datasette-cluster-map plugin can render these all on a map... but only if the columns have the names latitude and longitude.\nWe can use the datasette-edit-schema plugin to rename those columns.\nClick on the cog icon at the top of the table, then select \"Edit table schema\".\n\nRename the X and Y columns to longitude and latitude respectively, then click \"Apply changes\".\nReturn to the table page and you should see this map of Florida with manatee locations plotted on it:\n\nNext steps\nTry working through the rest of the Cleaning data with sqlite-utils and Datasette tutorial to learn more about how to use sqlite-utils and Datasette together."}, {"path": "/tutorials/data-analysis", "title": "Data analysis with SQLite and Python", "body": "Tutorials > Data analysis with SQLite and Python\n\nData analysis with SQLite and Python\nA 2h45m video tutorial about SQLite, Python, sqlite-utils and Datasette presented at PyCon 2023.\n\n\n\n\n    What you'll need\n\n    python3 and pip\n\n    Optional: GitHub Codespaces\n\n\n\n    sqlite-utils\n\n    Using the command-line tools to clean data\n\n\n\n    Exploring data with Datasette\n\n    Installing Datasette locally\n\n    Try a database: legislators.db\n\n    Install some plugins\n\n    Learning SQL with Datasette\n\n\n\n    Using sqlite-utils as a Python library, to import all the PEPs\n\n    Enabling full-text search\n\n    Publishing a database to Vercel\n\n    Other publishing options\n\n\n\n    Datasette Lite\n\n    Loading SQLite, CSV and JSON data\n\n    Installing plugins\n\n    Further reading\n\n\n\n    Advanced SQL\n\n    Aggregations\n\n    Subqueries\n\n    CTEs\n\n\n\n    JSON\n\n    Window functions\n\n\n\n    Baked Data\n\n    Niche Museums and TILs\n\n    Generating a newsletter with an Observable notebook\n\n\n\n    More demos and further reading\n\n    Fun demos\n\n    SpatiaLite\n\n\n\n\n\nWhat you'll need\npython3 and pip\nFor the first part of this tutorial, you'll need a Python 3 interpreter with the sqlite3 standard library module available.\nYou can run this on your own computer, or use a browser-based environment.\nhttps://pyodide.org/en/stable/console.html will work for a purely browser-based (WebAssembly) environment.\nFor the second part, you'll also need the ability to pip install Python packages.\nPython 3 on your own laptop (maybe in a fresh virtual environment) is a good option here. You could also use GitHub Codespaces or Google Colab or Jupyter Hub or your online notebook solution of choice.\nOptional: GitHub Codespaces\nI'll be working through the tutorial using GitHub Codespaces, using https://github.com/github/codespaces-jupyter\n\nsqlite-utils\nsqlite-utils provides \"CLI tool and Python utility functions for manipulating SQLite databases\".\nYou can install it the same way as Datasette:\npip install sqlite-utils\n\n\nOr with pipx:\npipx install sqlite-utils\n\n\nOr with Homebrew:\nbrew install sqlite-utils\n\n\nIt works as both a CLI tool and a Python library.\nUsing the command-line tools to clean data\nWe'll follow this tutorial next: Cleaning data with sqlite-utils and Datasette\nExploring data with Datasette\nDatasette is \"an open source multi-tool for exploring and publishing data\".\nInstalling Datasette locally\npip install datasette\n\n\nOr if you prefer pipx:\npipx install datasette\n\n\nOr Homebrew (on macOS):\nbrew install datasette\n\n\nMore installations options.\nIn Codespaces you should also install the datasette-codespaces plugin:\ndatasette install datasette-codespaces\n\n\nTry a database: legislators.db\nwget https://datasette.io/legislators.db\n\n\nThis is a database of US legislators, presidents and vice presidents.\nYou can explore it online at https://datasette.io/legislators\nOpen it in Datasette like this:\ndatasette legislators.db\n\n\nWe'll follow this tutorial to explore Datasette's features: Exploring a database with Datasette\nInstall some plugins\nDatasette has over a hundred plugins: https://datasette.io/plugins\nYou can pip install them, but it's better to use datasette install as that ensures they will go in the correct virtual environment, especially useful if you used pipx or Homebrew to install Datasette itself.\ndatasette install datasette-cluster-map\n\n\nNow restart Datasette and visit the \"offices\" table to see the result.\nYou can review what plugins are installed with:\ndatasette plugins\n\n\nOr by visiting the /-/plugins page in Datasette.\nPlugins can be uninstalled with:\ndatasette uninstall datasette-cluster-map\n\n\nLearning SQL with Datasette\nThe \"\u270e View and edit SQL\" link is a quick way to start learning basic SQL queries.\nWe'll follow this tutorial next: Learn SQL with Datasette\nUsing sqlite-utils as a Python library, to import all the PEPs\nLet's take our PEPs example from earlier and implement it again, but better, using sqlite-utils.\nI'll do this in a notebook.\n!git clone https://github.com/python/peps /tmp/peps\n\n\nWe now have ALL of the PEPs in /tmp/peps.\nimport pathlib\n\nfiles = list(pathlib.Path(\"/tmp/peps\").glob(\"pep-*.txt\"))\n\n\nAnd parse them with our function from earlier:\ndef parse_pep(s):\n    intro, body = s.split(\"\\n\\n\", 1)\n    pep = {}\n    current_key = None\n    current_value = None\n    for line in intro.split(\"\\n\"):\n        # If the line starts with whitespace, it's a continuation of the previous value\n        if line.startswith(\" \") or line.startswith(\"\\t\"):\n            if current_key is not None:\n                current_value += \" \" + line.strip()\n                pep[current_key] = current_value.strip()\n        else:\n            # Split the line into key and value\n            parts = line.split(\": \", 1)\n            if len(parts) == 2:\n                key, value = parts\n                # Update the current key and value\n                current_key = key\n                current_value = value\n                # Add the key-value pair to the pep dictionary\n                pep[current_key] = current_value.strip()\n    pep[\"Body\"] = body.strip()\n    return pep\n\n\npeps = []\nfor file in files:\n    peps.append(parse_pep(file.read_text()))\n\n\nWe now have a list of dictionaries. Let's load them into SQLite:\n%pip install sqlite-utils\n\n\nimport sqlite_utils\ndb = sqlite_utils.Database(\"/tmp/peps.db\")\ndb[\"peps\"].insert_all(peps, pk=\"PEP\", replace=True)\n\n\nI got this error:\nOperationalError: table peps has no column named PEP-Delegate\n\n\nTo fix that, use alter=True to automatically add any missing columns:\ndb[\"peps\"].insert_all(peps, pk=\"PEP\", alter=True, replace=True)\nprint(db[\"peps\"].count)\n# Outputs 429 \n\n\nEnabling full-text search\nSQLite has surprisingly good full-text search built in.\nsqlite-utils can help you enable it:\ndb[\"peps\"].enable_fts([\"Title\", \"Body\"])\n\n\nDatasette will detect this and add a search box to the top of the table page.\nTo run searches in relevance order you'll need to execute a custom SQL query:\nselect\n  PEP,\n  peps.Title,\n  Version,\n  Author,\n  Status,\n  Type,\n  Created,\n  peps.Body,\n  peps_fts.rank\nfrom\n  peps\njoin\n  peps_fts on peps.rowid = peps_fts.rowid\nwhere\n  peps_fts match :search\norder by\n  peps_fts.rank\nlimit\n  20\n\n\nPublishing a database to Vercel\nFirst, install both Vercel and the datasette-publish-vercel plugin.\nhttps://vercel.com/docs/cli has documentation for installing the Vercel CLI.\nOn macOS:\nbrew install vercel-cli\n\n\nOr use one of these:\nnpm i -g vercel\n\n\nOr:\npnpm i -g vercel\n\n\nNow run this command to login:\nvercel login\n\n\nInstall the plugin:\ndatasette install datasette-publish-vercel\n\n\nAnd deploy the database:\ndatasette publish vercel /tmp/peps.db --project python-peps\n\n\nOther publishing options\nDatasette can publish to the following providers:\n\nHeroku (datasette publish heroku)\nGoogle Cloud Run (datasette publish cloudrun)\nVercel (with datasette-publish-vercel)\nFly (with datasette-publish-fly)\n\nFurther deployment options are described in the documentation.\nDatasette Lite\nIt's Datasette... running entirely in your web browser with WebAssembly and Pyodide!\nhttps://lite.datasette.io/\nLoading SQLite, CSV and JSON data\n\nSQLite: https://lite.datasette.io/?url=https://github.com/simonw/scrape-hmb-traffic/blob/main/hmb.db?&install=datasette-copyable#/hmb?sql=with+item1+as+(%0A++select%0A++++time(datetime(commits.commit_at%2C+'-7+hours'))+as+t%2C%0A++++duration_in_traffic+%2F+60+as+mins_in_traffic%0A++from%0A++++item_version%0A++++join+commits+on+item_version._commit+%3D+commits.id%0A++order+by%0A++++commits.commit_at%0A)%2C%0Aitem2+as+(%0A++select%0A++++time(datetime(commits.commit_at%2C+'-7+hours'))+as+t%2C%0A++++duration_in_traffic+%2F+60+as+mins_in_traffic%0A++from%0A++++item2_version%0A++++join+commits+on+item2_version._commit+%3D+commits.id%0A++order+by%0A++++commits.commit_at%0A)%0Aselect%0A++item1.*%2C%0A++item2.mins_in_traffic+as+mins_in_traffic_other_way%0Afrom%0A++item1%0A++join+item2+on+item1.t+%3D+item2.t - see Measuring traffic during the Half Moon Bay Pumpkin Festival\nCSV: https://lite.datasette.io/?csv=https://raw.githubusercontent.com/fivethirtyeight/data/master/fight-songs/fight-songs.csv\nJSON: https://lite.datasette.io/?json=https://gist.github.com/simonw/73d15c0dd1025d1196829740bacf4464\n\nInstalling plugins\nAdd ?install=name-of-plugin to pip install that plugin into your browser's environment!\nThis only works with a subset of plugins.\n\nhttps://lite.datasette.io/?install=datasette-copyable&json=https://gist.github.com/simonw/73d15c0dd1025d1196829740bacf4464\n\nFurther reading\n\nDatasette Lite: a server-side Python web application running in a browser\nPlugin support for Datasette Lite\nJoining CSV files in your browser using Datasette Lite\n\nAdvanced SQL\nAggregations\nThe simplest form of aggregation is the one Datasette does to implement its own faceting feature.\nselect\n  party,\n  count(*)\nfrom\n  executive_terms\nwhere\n  type = 'prez'\ngroup by\n  party\n\n\nTry that query here.\nThe group by creates groups of rows, then the aggregation functions calculate a value across that entire group.\nThe most common aggregation functions are:\n\ncount(*) - count the number of rows in each group\nmax(column) - maximum value for a column\nmin(column) - minimum value for a column\nsum(column) - sum up the values in the column\n\nHere's an example of sum() and count() from What's in the RedPajama-Data-1T LLM training set:\nselect\n  top_folders,\n  sum(size_gb) as total_gb,\n  count(*) as num_files\nfrom raw\ngroup by top_folders\norder by sum(size_gb) desc\n\n\nRun that in Datasette Lite.\nChange the total_gb line to this to round it to the nearest integer:\n  cast (sum(size_gb) as integer) as total_gb,\n\n\nSubqueries\nSQLite has excellent support for subqueries. You can use them in where X in clauses:\nselect html_url from releases where repo in (\n  select id from repos where full_name in (\n    select repo from plugin_repos\n  )\n)\norder by created_at desc\n\n\nRun that on datasette.io. Sometimes I find these to be more readable than joins!\nYou can also use them directly in select clauses:\nselect\n  full_name,\n  (\n    select\n      html_url\n    from\n      releases\n    where\n      releases.repo = repos.id\n    order by\n      created_at desc\n    limit\n      1\n  ) as latest_release\nfrom\n  repos\n\n\nRun that here.\nCTEs\nCTE is a terrible name for an incredibly powerful feature. It stands for Common Table Expressions. Think of it as a way of creating an alias to a temporary table for the duration of a query.\nwith presidents as (\n  select\n    executives.name\n  from\n    executive_terms\n    join executives\n      on executive_terms.executive_id = executives.id\n  where\n    executive_terms.type = 'prez'\n),\nvice_presidents as (\n  select\n    executives.name\n  from\n    executive_terms\n    join executives\n      on executive_terms.executive_id = executives.id\n  where\n    executive_terms.type = 'viceprez'\n)\nselect\n  distinct name\nfrom\n  presidents\nwhere name in vice_presidents\n\n\nTry this CTE query here.\nJSON\nSQLite has excellent JSON functionality built in. Store JSON in a text column and you can query it using json_extract() - you can also build JSON values in select queries.\nReturning related rows in a single SQL query using JSON shows some advanced tricks you can do with this.\nselect\n  legislators.id,\n  legislators.name,\n  json_group_array(json_object(\n    'type', legislator_terms.type,\n    'state', legislator_terms.state,\n    'start', legislator_terms.start,\n    'end', legislator_terms.end,\n    'party', legislator_terms.party\n   )) as terms,\n   count(*) as num_terms\nfrom\n  legislators join legislator_terms on legislator_terms.legislator_id = legislators.id\n  group by legislators.id\norder by\n  id\nlimit\n  10\n\n\nRun that query.\nPaul Ford said about SQLite's JSON support:\n\nThe JSON interface is like, \u201cwe save the text and when you retrieve it we parse the JSON at several hundred MB/s and let you do path queries against it please stop overthinking it, this is filing cabinet.\u201d\n\nWindow functions\nI wanted to run a query that would return the following:\n\nThe repository name\nThe date of the most recent release from that repository (the releases table is a many-to-one against repos)\nThe total number of releases\nThe three most recent releases (as a JSON array of objects)\n\nwith cte as (\n  select\n    repos.full_name,\n    releases.created_at,\n    releases.id as rel_id,\n    releases.name as rel_name,\n    releases.created_at as rel_created_at,\n    rank() over (partition by repos.id order by releases.created_at desc) as rel_rank\n  from repos\n    left join releases on releases.repo = repos.id\n)\nselect\n  full_name,\n  max(created_at) as max_created_at,\n  count(rel_id) as releases_count,\n  json_group_array(\n    json_object(\n      'id', rel_id,\n      'name', rel_name,\n      'created_at', rel_created_at\n    )\n  ) filter (where rel_id is not null and rel_rank <= 3) as recent_releases\nfrom cte\ngroup by full_name\norder by releases_count desc\n\n\nRun that query here.\nRunning this smaller query helps show what's going on with that rel_rank column:\nwith cte as (\n  select\n    repos.full_name,\n    releases.created_at,\n    releases.id as rel_id,\n    releases.name as rel_name,\n    releases.created_at as rel_created_at,\n    rank() over (partition by repos.id order by releases.created_at desc) as rel_rank\n  from repos\n    left join releases on releases.repo = repos.id\n)\nselect * from cte\n\n\nBaked Data\nThe Baked Data architectural pattern describes this approach, which is key to taking full advantage of SQLite and Datasette.\nI like to build my databases in GitHub Actions.\nNiche Museums and TILs\n\nhttps://www.niche-museums.com/ is published from the https://github.com/simonw/museums repository - one big YAML file for the content.\nhttps://til.simonwillison.net/ is published https://github.com/simonw/til - separate Markdown files for each item.\n\nBoth of these sites have Atom feeds that are defined using a Datasette canned query, in conjunction with the datasette-atom plugin.\n\nhttps://www.niche-museums.com/browse/feed\nhttps://til.simonwillison.net/tils/feed\n\nGenerating a newsletter with an Observable notebook\nI wrote about this in Semi-automating a Substack newsletter with an Observable notebook:\n\nhttps://datasette.simonwillison.net/simonwillisonblog is a Datasette/SQLite copy of my Django blog, created using db-to-sqlite by my https://github.com/simonw/simonwillisonblog-backup GitHub repository.\nhttps://observablehq.com/@simonw/blog-to-newsletter is my Observable notebook that assembles a newsletter from that data.\nhttps://simonw.substack.com/ is the Substack newsletter that I copy that content into.\n\nMore demos and further reading\nFun demos\nI post a lot of Datasette projects on my blog. Some of my recent favourites:\n\nExploring MusicCaps, the evaluation data released to accompany Google\u2019s MusicLM text-to-music model shows how Datasette can be used to explore data used as part of training a machine learning model.\nAnalyzing ScotRail audio announcements with Datasette\u2014from prototype to production explains the scotrail.datasette.io project.\nI built a ChatGPT plugin to answer questions about data hosted in Datasette is one of my earlier explorations at the intersection of LLM AI and Datasette.\n\nSpatiaLite\n\nBuilding a location to time zone API with SpatiaLite shows how to use SpatiaLite and Datasette to create an API for looking up time zones based on latitude/longitude points.\nDrawing shapes on a map to query a SpatiaLite database introduces the datasette-leaflet-freedraw plugin and introduces the calands.datasettes.com demo.\nJoining CSV and JSON data with an in-memory SQLite database"}, {"path": "/tutorials/explore", "title": "Exploring a database with Datasette", "body": "Tutorials > Exploring a database with Datasette\n\nExploring a database with Datasette\nThis tutorial will teach you how to use Datasette to explore a new database.\nDatasette offers flexible tools for exploring data tables. It's always worth spending time familiarizing yourself with data in its raw, tabula form before thinking about ways to apply more sophisticated analysis or visualization.\nThe congress-legislators data\nWe'll be using an example database of Members of the United States Congress, 1789 to present. I built this example using data from the unitedstates/congress-legislators project on GitHub, maintained by Joshua Tauberer, Eric Mill and over 100 other contributors.\nAccess the example in Datasette here: https://datasette.io/legislators\n\nUnderstanding tables\nThe key to understanding relational databases such as SQLite (which we are using here), MySQL, PostgreSQL, Oracle and so on is to learn to think in tables.\nDatabase tables are like spreadsheets - they have columns and rows, and each row provides a value for every column. Unlike spreadsheets you can't merge cells together or apply formatting - a database table is a straight-forward two-dimensional grid. It's just the data.\nEach column in a table has a type. In SQLite these types can be one of text, integer, real (for floating point values) and blob (for binary data). The type of the column matters because it affects what happens when you sort the table by that column, or which mathematical operations you can use against those values.  \nBrowsing tables\nThe tables in our example database are:\n\nlegislators - over 12,000 rows - individuals who have served as senators or representatives\nlegislator_terms - over 45,000 rows - terms served by legislators, back to 1789\nexecutives - 80 rows - individuals who have served as President or Vice President\nexecutive_terms - over 130 rows - terms served by executives\noffices - over 1,300 rows - district officers maintained by current legislators\nsocial_media - over 500 rows - social media accounts for current legislators\n\nFollow these links now and explore the tables to get a feel for the way the data is organized in this database.\nTables can relate to each other\nSometimes you'll see a row in a table that links to another table, for example the legislator_id column in the legislator_terms table:\n\nClicking one of the linked names in that column will take you to a page for that individual legislator row that looks like this:\n\nThe \"Links from other tables\" section shows how many rows in other tables refer back to this legislator.\nIn a database these are called \"foreign keys\" - they work by storing the ID of a row from another table in a dedicated foreign key column.\nForeign keys are key to understanding why relational databases are so much more powerful than separate, stand-alone data files stored in something like CSV.\nUsing facets\nFacets are one of the most powerful Datasette features. They can help you take a table with thousands of rows and start quickly identifying interesting trends and patterns within that data.\nI'll show some examples using the executive_terms table.\nFacets can be applied in two ways: you can select an option from the \"suggested facets\" list, or you can select the \"Facet by this\" option from the cog menu next to each column.\n\nOnce selected, the facet interface is shown above the table, like in this example:\n\nEach facet shows a list of the most common values for that column, with a total count number for each of those values.\nThese numbers will update as you further filter the data: if you select \"Democratic\" you will see just the rows matching that party, and the \"type\" facet will update to show you that there have been 21 presidentals terms and 19 vice presidental terms for that party.\n\nExercises\n\nApply the type, state and party facets to the legislator_terms table\nUse them to filter just to Republican Senators from North Carolina (NC)\nFigure out which state has had the highest number of senatorial terms held by a member of the Republican party. (solution if you get stuck)\n\nUsing filters\nWhen you select a facet, you're applying a filter to the data. These are reflected in the boxes at the top of the page:\n\nYou don't have to use faceting for these - you can edit them directly.\nTo see Democratic Vice Presidents just for the 1800s, we can add a filter for rows where the 'start' column begins with '18':\n\nHere's the result of that query.\n\nExercises\nUsing the legislator_terms table, find:\n\nEvery senatorial term served in Delaware (DE)\n... that started in a year starting with 18\n... and then facet by party to see which parties had the most senatorial terms\n\nSharing links\nEvery page within Datasette is designed to be shared. Copy and paste the page URL to share it with someone else.\nThis includes applied filters and facets - so if you find an interesting way to explore the data you can share that directly by sharing the URL.\nSome examples:\n\nEvery senatorial term in the state of Texas\nEvery executive term held by a member of the Democratic-Republican party\n\nExercise: share some links!\nUse the above exercises to find something interesting in the data. Copy and paste that URL to your notes, or share it with a friend, or post about it on Twitter.\nExporting data\nIf you can see it in Datasette, you can export the raw data. This is a fundamental principle of the project.\nLook for the CSV and .json links on any page to export the data in those formats.\n\nThe \"Advanced export\" box provides additional options. Use \"download file\" to download the CSV file to your computer. \nThe \"expand labels\" option will add an extra column with the label associated with any foreign key columns - in this example, it will add legislator_id_label to accompany the ID in legislator_id.\nAdditional formats can be enabled using plugins. The example instance is running the datasette-copyable plugin, which provides a `\"copyable\" link which can be used to export the data in different formats using copy-and-paste!\nDatasette can also be configured to allow users to download the entire database as a single file. You can find that legislators.db download option at the bottom of https://datasette.io/legislators - the entire database is around 5.5MB.\nExercises\n\nDownload a CSV file of Democratic Vice Presidents in the 1800s, using the \"download file\" checkbox in the \"Advanced export\" box.\nUse the \"copyable\" option to copy and paste that data directly into Google Sheets, Excel or Numbers.\n\nData visualization via plugins\nYou may have noticed this already on the offices table, which shows an interactive map of all of the office locations:\n\nYou can click on clusters of markers to zoom in, and click on individual markers to see the details of the row.\nIt's fun to use these maps to spot obvious outliers - in this case, the office of representative Aumua Amata Radewagen in American Samoa instantly stands out.\nThis is another example of a Datasette plugin - in this case datasette-cluster-map. Plugins can add additional features to Datasette, such as visualizations or alternative export methods (see \"copyable\" described above).\nFor a larger example of a map visualization, take a look at this map of 33,000+ global power plants visualized using datasette-cluster-map.\nNext steps\nNow that you know how to explore tables using Datasette, the next covers how to use Datasette to learn SQL."}, {"path": "/tutorials/learn-sql", "title": "Learn SQL with Datasette", "body": "Tutorials > Learn SQL with Datasette\n\nLearn SQL with Datasette\nThis tutorial follows the tutorial on exploring data with Datasette and shows how you can use Datasette to start learning to write custom SQL queries of your own.\nWe will be using the same example database as that tutorial:\n\nhttps://datasette.io/legislators\n\nThis database contains information about US Presidents, Vice Presidents and Members of Congress from 1789 to today. You can follow the previous tutorial to become familiar with the data.\nSQL and SQLite\nDatasette is software that runs on top of a SQL database. There are many different SQL database systems, such as MySQL, PostgreSQL, Microsoft SQL Server and Oracle. The database used by Datasette is called SQLite.\nYou may not realize it, but you already use SQLite every day: it is built into many popular applications, including Google Chrome and Firefox, and runs on laptops, iPhones, Android phones and all sorts of other smaller devices.\nSQL stands for Structured Query Language - it is a text language for running queries against a database. Every database implements a slightly different dialect of SQL - for this tutorial I will try to keep to the subset of SQLite's dialect that is most likely to work across other databases as well.\nView and edit SQL\nEvery table page in Datasette - such as this one - includes a \"View and edit SQL\" link that looks like this:\n\nClick that link to see the SQL query that was used for the current page - including any filters that have been applied, in this case the filter for bio_gender = 'F'.\nYou can then edit that query to make it do something else!\n\nExercises\nStart with that example, then:\n\nClick \"Format SQL\" to tidy this up and make it easier to read and edit\nModify that query to return only the id and name columns\nGet it to order by name instead of sorting by id\nChange the p0 value to M to return male instead of female legislators\nTry replacing the list of columns with select * from - a shortcut for all columns\n\nThe structure of a basic select query\nHere's the formatted version of the above query:\nselect\n  id,\n  name,\n  id_bioguide,\n  id_govtrack,\n  id_icpsr,\n  id_wikipedia,\n  id_wikidata,\n  id_google_entity_id,\n  name_first,\n  name_last,\n  bio_birthday,\n  bio_gender,\n  id_house_history,\n  name_middle,\n  name_nickname,\n  id_ballotpedia,\n  name_suffix,\n  id_bioguide_previous,\n  id_house_history_alternate,\n  other_names,\n  id_thomas,\n  id_cspan,\n  id_votesmart,\n  id_lis,\n  name_official_full,\n  id_opensecrets,\n  id_fec,\n  id_maplight,\n  leadership_roles,\n  family\nfrom\n  legislators\nwhere\n  bio_gender = :p0\norder by\n  id\nlimit\n  101\n\nselect\nThe select section specifies which columns you would like to return. Each column name is separated by a comma - but if you have a comma following the last column you you will get an error message.\n\nnear  from: syntax error\n\nfrom\nThe from section specifies which table the records should be selected from - here we want the legislators table.\nwhere\nThe where section adds filter conditions. These can be combined using and, for example this query will select just legislators who are male and have a Jr. name suffix:\nwhere\n  bio_gender = 'M'\n  and name_suffix = 'Jr.'\n\nThis section is optional - if you do not include a where clause you will get back every row in the table.\norder by\nThe optional order by clause specifies the order you woud like the rows to be returned in. This would order them alphabetically by name:\norder by\n  name\n\nOr add desc to reverse the order. This query returns the youngest legislators:\nselect\n  *\nfrom\n  legislators\norder by\n  bio_birthday desc\n\nlimit\nThe limit 101 clause limits the query to returning just the first 101 results. In most SQL databases omitting this will cause all results will be returned - but Datasette applies an additional limit of 1,000 (example here) to prevent large queries from causing performance issues.\nNamed parameters\nIn the previous tutorial we used filters to list presidental terms that occurred in the 1800s, by filtering on rows where the start column began with the string 18 and the type column equals prez.\nHere's that example as a filtered table.\n\nClicking that \"View and edit SQL\" link (and then clicking \"Format SQL\") shows this query:\n\nselect\n  rowid,\n  type,\n  start,\n  end,\n  party,\n  how,\n  executive_id\nfrom\n  executive_terms\nwhere\n  start like :p0\n  and type= :p1\n\nThe :p0 and :p1 fields here correspond to \"named parameters\" - they provide a safe way to pass values to a SQL query, without having to worry about SQL injection.\nThese field names are extracted from the query. If you change the where clause to look like this:\nwhere\n  start like :century\n  and type = :type\n\nThen fields called century and type will be displayed instead of p0 and p1.\nSince they are form fields, you can change those values too. Try changing prez to viceprez to see the vice presidential terms for the 1800s instead.\nIf you weren't using named parameters, the where clause of the query could look like this instead:\nwhere\n  start like '18%'\n  and type = 'prez'\n\nString values like '18%' and 'prez' must be enclosed in single or double quotes (single quotes are preferred), if you are not using the :p0 syntax.\nThe double quotes around the column names here are optional - they are only required if the column has a name that might clash with an existing SQL keyword, such as select or where - or if the column name contains a space. So the following where clause would work the same way:\nwhere\n  start like '18%'\n  and type = 'prez'\n\nSQL LIKE queries\nTo find rows where the start column begins with the string 18 we use this where filter:\nwhere\n  start like '18%'\n\nThe SQL like operator applies wildcards to strings - % means \"match anything\" and _ (underscore) means \"match a single character\".\nExercise\n\nUse a LIKE filter to find presidential terms that occurred in the 90s decade of any century - you'll need to use both _ and % for this. (solution)\n\nSQL Joins\nThe executive_terms table has a column called executive_id which displays both a numeric ID and a link containing the name of the executive:\n\nBut... when you click the \"View and edit SQL\" link the resulting query returns just the ID, not the name.\n\nA SQL join can be used to combine the data from multiple tables. Here's a query that uses a join to show the name pulled from the executives table:\nselect\n  executive_terms.type,\n  executive_terms.start,\n  executive_terms.end,\n  executive_terms.party,\n  executive_terms.how,\n  executives.name\nfrom\n  executive_terms\n  join executives\n    on executive_terms.executive_id = executives.id\n\nTry that here.\n\nSince there are now two tables involved in the query, it's a good idea to include explicit table names as part of the select clause:\nselect\n  executive_terms.type,\n  executive_terms.start,\n  executive_terms.end,\n  executive_terms.party,\n  executive_terms.how,\n  executives.name\n\nIf a column exists in only one of the two tables you can use its name without specifying the table, but this can quickly get confusing so it's better to always use the table names when you are executing a join.\nThe from clause is where the join is defined. This describes how the two tables should be combined together:\nfrom\n  executive_terms\n  join executives\n    on executive_terms.executive_id = executives.id\n\nWe are joining executive_terms to executives, using the fact that the executive_id column in executive_terms contains values from the id column in executives.\nThis is why IDs and foreign keys are important concepts to understand in SQL databases!\nThis kind of join is also known as an \"inner join\" - it is the most commonly used join. Other join types include outer joins and full joins, but those are beyond the scope of this tutorial.\nGROUP BY / COUNT\nA common operation in SQL is to ask for a count of the most popular values in a column.\nDatasette exposes this capability in its faceting feature. Under the hood, that feature works by executing a group by / count query.\nThe following query answers the question: which party has had the most presidental terms?\nselect\n  party,\n  count(*)\nfrom\n  executive_terms\nwhere\n  type = 'prez'\ngroup by\n  party\n\nTry that query here.\n\nThe select clause asks for the party column and the result of running a count(*). count() is a SQL aggregate function, which operates on a group of results. So we need to define a group.\nThe group by party clause at the end creates those groups. The combination of the two produces the desired result.\nYou can add this order by clause at the end to see the groups sorted from highest to lowest:\norder by\n  count(*) desc\n\nHere's the result.\nWHERE ... IN\nSQL queries can be nested together in a number of interesting ways.\nOne of the most useful for ad-hoc data analysis is the where column in (select ...) pattern.\nLet's build a query to see everyone who has been president and has also been vice president.\nWe'll start with a query to return the names of every vice president. This requires a join against executives, because the executive_terms table includes the presidential and vice presidential terms, but doesn't include the names of those individuals.\nselect\n  executives.name\nfrom\n  executive_terms\n  join executives\n    on executive_terms.executive_id = executives.id\nwhere\n  type = 'viceprez'\n\nThis query returns 61 rows, including some duplicate rows for individuals who served more than one term as VP.\nWe could use select distinct to remove those duplicates if we wanted to.\nIf we want to see everyone who has been a president and ALSO been a vice-president, we can combine two queries together like this:\nselect\n  distinct executives.name\nfrom\n  executive_terms\n  join executives\n    on executive_terms.executive_id = executives.id\nwhere\n  type = 'prez'\n  and name in (\n    select\n      executives.name\n    from\n      executive_terms\n      join executives\n        on executive_terms.executive_id = executives.id\n    where\n      type = 'viceprez'\n  )\n\nThis query returns 15 names, starting with John Adams and finishing with Joseph Biden.\n\n\u26a0\ufe0f An earlier version of this example revealed a subtle bug: there were two separate individuals in the executives table with the same name, George Bush and George Bush! So even though the query appeared to return the correct results it actually contained a bug. This is a useful demonstration of how important it is to work with unique, de-duplicated identifiers where-ever possible, rather than assuming that things like names are unique.\n\nCommon Table Expressions\nThis is a more advanced SQL technique that I find myself using every day: it can make complex SQL queries much easier to write and understand.\nCommon Table Expressions, or CTEs, allow you to define a temporary alias for a select that lasts for the duration of the current query. You can then treat it as if it was a regular table.\nHere's an example that creates two CTEs, one called presidents and one called vice_presidents, and then uses them to answer the earlier question about presidents who had also served as vice presidents:\nwith presidents as (\n  select\n    executives.name\n  from\n    executive_terms\n    join executives\n      on executive_terms.executive_id = executives.id\n  where\n    executive_terms.type = 'prez'\n),\nvice_presidents as (\n  select\n    executives.name\n  from\n    executive_terms\n    join executives\n      on executive_terms.executive_id = executives.id\n  where\n    executive_terms.type = 'viceprez'\n)\nselect\n  distinct name\nfrom\n  presidents\nwhere name in vice_presidents\n\nTry this CTE query here.\nThe two CTEs are defined using with alias_name as (select ...), second_alias as (select ...) - then the query that returns the final results is added on at the end.\nCreating bookmarkable apps\nSince every page on Datasette can be linked to (see Sharing links) and named parameters automatically add form input to the query page, you can combine these two features to create bookmarkable apps.\nThis is particularly useful for collaborating with others who don't yet know SQL. You can write SQL queries that solve problems for them, add parameters that they can customize and then send them the entire custom application as a link that they can bookmark.\nLet's create an application that returns the president and vice president for any given date.\nHere's the SQL query we will be using:\nselect\n  executives.name,\n  executive_terms.type,\n  executive_terms.start,\n  executive_terms.end,\n  executive_terms.party\nfrom\n  executive_terms\n  join executives\n    on executive_terms.executive_id = executives.id\nwhere\n  start <= :date\n  and end > :date\n\nThis query takes a :date parameter, which should be of the format yyyy-mm-dd - for example 2016-01-01.\nIt joins against the executives table to get their name, and then filters to rows where the term start is less than or equal to the specified date, and the end is greater than that date. We use <= for one of these to ensure there are no gaps or overlaps.\nIf we execute the query without a date it will return 0 results, but give us a form field to enter the date:\n\nEntering a date - for example 2016-01-01 - returns the president and vice president for that date:\n\nWe've built an application! We can send this link to anyone right now and they'll be able to run that same query.\nThe SQL at the top of the page could feel a little intimidating though. That's what the \"hide\" link is for - clicking on that hides the SQL query, providing a new link that you can share that won't cause the user to scroll down past the SQL in order to interact with the query.\nOne last note: the application we have built also doubles up as an API. Add .json to the path portion of the URL, or click the .json link, to get back the data as JSON. Add &_shape=array to the URL to get back a more compact form of JSON, ideal for integrating with other applications.\nNext steps\nFully mastering SQL can take years. This tutorial has attempted to cover SQL basics and introduce some more advanced techniques to help manage larger queries, but there's plenty more to learn. The SQLite SELECT documentation offers a comprehensive reference to SQL as understood by SQLite, but the best way to learn is to keep on trying new things against different data, looking things up as you go along."}, {"path": "/tutorials/spatialite", "title": "Building a location to time zone API with SpatiaLite", "body": "Tutorials > Building a location to time zone API with SpatiaLite\n\nBuilding a location to time zone API with SpatiaLite\nThe SpatiaLite extension for SQLite adds a large number of functions for geospatial analysis, which can be used with Datasette to build GIS (Geographic Information System) applications.\nThis tutorial will show how SpatiaLite and Datasette can be combined to create a JSON API that can return the time zone for a specific latitude and longitude point on earth.\nWhat we are going to build\nYou can try the API out here. Give it a latitude and longitude and it will return the corresponding time zone ID: https://timezones.datasette.io/timezones/by_point\nSome examples:\n\n\nBrighton, England is in Europe/London (in JSON)\n\nSan Francisco, USA is in America/Los_Angeles (in JSON)\n\nTokyo, Japan is Asia/Tokyo (in JSON)\n\nSetting up a development environment\nYou will need two things in place for this tutorial:\n\nSpatiaLite installed on your system\nA Python installation that allows the sqlite3 module to load additional extensions\n\nRecommended: Use GitHub Codespaces\nGitHub Codespaces can provide you with a free development environment for this project accessible through your web browser, with all of the tools you need pre-installed.\nThis is the easiest way to work through this tutorial.\nVisit this link to create a new Codespace with everything you will need for the rest of the tutorial.\nUsing a Mac\nOn a Mac you can install SpatiaLite and Datasette using Homebrew like so:\nbrew install spatialite-tools datasette\n\nIf you are on a Mac you may find that your installation of Python cannot load external SQLite modules. You can check by running the following:\ndatasette --load-extension spatialite\n\nIf you get an error message about enable_load_extension then consult this page for hints on how to fix the problem.\nBuilding the database\nTo build this project we first need geographic shapes for all of the world's time zones.\ntimezone-boundary-builder is a project by Evan Siroky which uses data from OpenStreetMap to create detailed time zone polygons, which he then releases in GeoJSON and Shapefile format. The result is made available under the Open Data Commons Open Database License (ODbL).\nStart by downloading the timezones-with-oceans.shapefile.zip file from the latest release:\nwget https://github.com/evansiroky/timezone-boundary-builder/releases/download/2022g/timezones-with-oceans.shapefile.zip\n(If you are using Codespaces you should run all of these commands in the \"Terminal\" tab of the Codespaces interface.)\nA Shapefile is a set of files that describe a collection of geographic features. There's no need to unzip this zip file - our tools can work directly with it.\nWe can load the file into a new SpatiaLite database using the shapefile-to-sqlite command.\nIf you are running this tutorial in Codespaces this has been installed already. Otherwise, you can install it like this:\npip install shapefile-to-sqlite\n\nTo load the Shapefile into a database, use the following:\nshapefile-to-sqlite timezones.db \\\n    timezones-with-oceans.shapefile.zip \\\n    --table timezones \\\n    --spatial-index\n\nThis will create a new database file called timezones.db, and load the shapes from the Shapefile into a new table called timezones. It will also set up a spatial index on the timezones table, described later.\nThe command will show a progress bar like this:\nzip://timezones-with-oceans.shapefile.zip\n  [########----------------------------]   22%  00:00:38\n\nThis make take a couple of minutes to complete.\nBrowsing the database in Datasette\nWe can now start Datasette against that file to browse the data:\ndatasette timezones.db --load-extension spatialite\n\nYou should see the following output:\nINFO:     Started server process [5385]\nINFO:     Waiting for application startup.\nINFO:     Application startup complete.\nINFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)\n\nIf you are using your own computer you can now visit http://127.0.0.1:8001 to see the Datasette interface.\nIf you are running in Codespaces the tool should offer an \"Open in Browser\" button.\nYou can now browse the database. There should be a timezones table with three columns: id, tzid and geometry\nThere isn't much to see yet though! The geometry columns are just big binary blobs.\nViewing the time zone shapes on a map\nTo see the time zone geometries on a map, we can install a Datasette plugin.\ndatasette-geojson-map by Chris Amico adds the ability to render GeoJSON and SpatiaLite geometries directly on a map.\nHit Ctrl+C in the terminal to stop the Datasette server, then run the following:\ndatasette install datasette-geojson-map\n\nNow start Datasette running again:\ndatasette timezones.db \\\n  --load-extension spatialite \\\n  --setting default_page_size 10\n\nWe're adding an extra setting here, to set the default page size to 10. This is because some of the time zone polygons are really large and the default page size of 100 can take a long time to render.\nVisit the timezones table again and you should see something like this:\n\nFinding the time zone for a point\nNow that we have a table full of time zone geometries we can construct a SQL query that can show us the time zone for any specific point on earth.\nWe can do this using the SpatiaLite within() function, which takes two geometries and checks if one is contained within the other.\nFirst we're going to need a geometry that represents a specific latitude/longitude point.\n41.798, -87.696 is a point within the city of Chicago - see it here on Google Maps.\nWe can use the SpatiaLite MakePoint(longitude, latitude) function (note that the latitude and longitude are reversed here) to create a geometry which can then be used with the within() function:\nselect\n  tzid\nfrom\n  timezones\nwhere\n  within(\n    MakePoint(-87.696, 41.798),\n    timezones.Geometry\n  ) = 1\nwithin(geom1, geom2) returns 1 if the first geometry is contained within the second, and 0 if it is not.\nSure enough, this query returns America/Chicago.\nUsing parameters in the SQL query\nDatasette has a feature where SQL queries can contain named parameters :like_this which will be turned into form fields and used to feed new values to the queries.\nTry that with the following query:\nselect\n  tzid\nfrom\n  timezones\nwhere\n  within(\n    MakePoint(:longitude, :latitude),\n    timezones.Geometry\n  ) = 1\nThis looks like it should work... but if you try it with the previous coordinates you'll see that it returns no results.\nThis is because MakePoint() needs to be given floating point values, but Datasette passes all parameters as strings.\nIf you pass invalid types, MakePoint() returns null - and within(null, geometry) then returns -1 to specify an invalid result.\nWe can fix this by casting the strings to floats, like this:\nselect\n  tzid\nfrom\n  timezones\nwhere\n  within(\n    MakePoint(cast(:longitude as float), cast(:latitude as float)),\n    timezones.Geometry\n  ) = 1\nThis query has the desired effect: given a latitude and longitude for any point on earth it will return the correct time zone.\nSpeeding it up with an index\nThere's one remaining catch with this query: it's relatively slow. In my testing I was seeing anything between 150ms and 750ms for the query to run, due to the need to compare the point with all 455 polygons in the database.\nWe used the --spatial-index option when we first imported the data. Here's how to take advantage of that spatial index to speed things up:\nselect tzid\nfrom\n  timezones\nwhere\n  within(\n    MakePoint(cast(:longitude as float), cast(:latitude as float)),\n    timezones.Geometry\n  ) = 1\n  and rowid in (\n    select\n      rowid\n    from\n      SpatialIndex\n    where\n      f_table_name = 'timezones'\n      and search_frame = MakePoint(cast(:longitude as float), cast(:latitude as float))\n  )\nThe trick here is the extra and rowid in (...) subquery.\nSpatialIndex is a special table - it's a virtual table created by SpatiaLite.\nYou can query that table by passing the name of another table that has a spatial index associated with it - in this case timezones. Then you pass in a geometry as search_frame and the table will return a list of rowid values representing any polygons with a rough bounding box that overlaps that of the geometry you passed in.\nNote that this is not an exact comparison: some of the row IDs you get back may not intersect exactly with the geometry.\nBut it's a good enough approximation. You can combine these values with a more accurate within() check, which will then only have to run full calculations against a subset of the overall set of polygons.\nIn my local testing this dropped the time taken for the query from 150ms to less than 8ms - a significant speedup!\nAdding country polygons\nWhich time zones are relevant for a specific country?\nOur data at the moment can't tell us that. We can filter for just time zones that start with America/ but this will give us everything for both North and South America. Wouldn't it be neat if we could browse our time zones by country instead?\nWe can try to answer that question by loading in polygons for every country in the world.\ndatahub.io/core/geo-countries has the data we need for this, derived from Natural Earth and released under the PDDL license.\nWe can download a GeoJSON file of countries like this:\nwget https://datahub.io/core/geo-countries/r/countries.geojson\n\nNow we can use geojson-to-sqlite to load it into a countries database table, again creating a spatial index:\npip install geojson-to-sqlite\n\ngeojson-to-sqlite timezones.db countries \\\n  countries.geojson --spatial-index \n\nDatasette should pick up the new countries table, and datasette-geojson-map will now show rendered outlines of those countries.\nTo find the time zones that intersect a specific country, we can use the following query:\nselect\n  tzid,\n  geometry\nfrom\n  timezones\nwhere intersects(\n  timezones.geometry, (\n    select geometry from countries where admin = 'United Kingdom'\n  ) = 1\n)\nWe are using a couple of additional tricks here.\nThe intersects() function is a SpatiaLite function that checks if two geometries intersect each other in any way.\nWe'e also using a subquery to access the geometry of the country that we are interested in:\nselect geometry from countries where admin = 'United Kingdom'\nThe admin column in countries has names of the countries - here we are pulling back the geometry for the United Kingdom.\nHere are the results of this query.\nSimplifying the polygons\nTry changing the country name to \"United States of America\" and you may run into a problem: the geometries for the time zones that intersect with the United States are so large that they may not be possible to render on the map!\nWe can help solve that using another SpatiaLite function: simplify(). This function applies the Douglas\u2013Peucker algorithm to simplify a polygon down to a smaller number of points, which makes it much easier to render.\nsimplify() takes a geometry and a precision value. After some trial and error I found that a precision value of 0.05 worked well for these time zone polygons:\nselect\n  tzid,\n  simplify(geometry, 0.05) as geometry\nfrom\n  timezones\nwhere intersects(\n  timezones.geometry, (\n    select geometry from countries where admin = 'United States of America'\n  )  = 1\n)\nIt's necessary to use simplify(...) as geometry here because the datasette-geojson-map plugin will look for an output column called geometry to render on the map.\nHere's the results of this query for the United States.\nSpeeding that up with an index\nWe can use a spatial index in a similar way to the time zone query listed above:\nwith country as (\n  select\n    geometry\n  from\n    countries\n  where\n    admin = :country\n)\nselect\n  timezones.tzid,\n  simplify(timezones.geometry, 0.05) as geometry\nfrom\n  timezones,\n  country\nwhere\n  timezones.id in (\n    select\n      SpatialIndex.rowid\n    from\n      SpatialIndex,\n      country\n    where\n      f_table_name = 'timezones'\n      and search_frame = country.geometry\n  )\n  and intersects(timezones.geometry, country.geometry) = 1\nThis uses the same trick as before - a where timezones.id in (...) subselect that returns a list of likely rowid values from the SpatialIndex virtual table.\nIf a table has an integer primary key - such as our timezones table here - SQLite will set the rowid to be the same value, which is why we can compare timezones.id with rowid in this query.\nWe're using one more trick here. To avoid having to run that query to select the country geometry twice, we're instead bundling that into a CTE - a Common Table Expression - at the start of the query.\nThe with country as (...) piece makes the result of that query available as a temporary table called country for the duration of the SQL query.\nDefining metadata with canned queries\nNow that we've figured out all of the queries needed to power our API, it's time to tie them together into a configuration that we can deploy to the internet.\nDatasette's metadata system can be used to provide extra information about the database, and it can also be used to configure canned queries - SQL queries with names, like the by_point example shown at the beginning of this tutorial.\nHere's a metadata.yml file which defines both the by_point query and a query for looking up time zones by country code.\ntitle: Time zones API\ndescription: |\n  An API for looking up time zones by latitude/longitude\nabout: simonw/timezones-api\nabout_url: https://github.com/simonw/timezones-api\nlicense: ODbL\nlicense_url: http://opendatacommons.org/licenses/odbl/\nsource: timezone-boundary-builder\nsource_url: https://github.com/evansiroky/timezone-boundary-builder\nallow_sql: false\ndatabases:\n  timezones:\n    tables:\n      countries:\n        source: Natural Earth\n        source_url: https://www.naturalearthdata.com/\n        license: Open Data Commons Public Domain Dedication and License (PDDL) v1.0\n        license_url: https://opendatacommons.org/licenses/pddl/1-0/\n        about: geo-countries\n        about_url: https://datahub.io/core/geo-countries\n    queries:\n      by_point:\n        title: Find time zone by lat/lon\n        sql: |\n          select tzid\n          from\n            timezones\n          where\n            within(\n              MakePoint(cast(:longitude as float), cast(:latitude as float)),\n              timezones.Geometry\n            ) = 1\n            and rowid in (\n              select\n                rowid\n              from\n                SpatialIndex\n              where\n                f_table_name = 'timezones'\n                and search_frame = MakePoint(cast(:longitude as float), cast(:latitude as float))\n            )\n      by_country:\n        title: Find time zones that intersect a country\n        sql: |\n          with country as (\n            select\n              geometry\n            from\n              countries\n            where\n              admin = :country\n          )\n          select\n            timezones.tzid,\n            simplify(timezones.geometry, 0.05) as geometry\n          from\n            timezones,\n            country\n          where\n            timezones.id in (\n              select\n                SpatialIndex.rowid\n              from\n                SpatialIndex,\n                country\n              where\n                f_table_name = 'timezones'\n                and search_frame = country.geometry\n            )\n            and intersects(timezones.geometry, country.geometry) = 1\nIn addition to providing canned queries called by_name and by_country, this file also includes metadata showing the source of the data we used for the database.\nIt also sets one more important option:\nallow_sql: false\nThis option prevents users from executing their own custom SQL queries against our published database. Only the canned queries we have defined will be available.\nWe're using that option here because SpatiaLite has a large number of functions, some of which could crash the underlying Datasette instance.\nWe can test our new metadata.yml file by starting Datasette like this:\ndatasette timezones.db --load-extension spatialite -m metadata.yml\n\nDeploying the application to Fly\nThe datasette publish command can be used to deploy Datasette instances to a variety of different hosting providers.\nFly is an excellent choice for hosting this, since the API could attract a large amount of traffic.\nGoogle Cloud Run charges based on how much sustained traffic an instance gets, which could become expensive for this application.\nFly charge a flat monthly rate for the instance, plus additional fees for bandwidth.\nCurrent Fly pricing can be found here. At the time of writing an instance with 256MB of RAM - enough to comfortably host this API - costs $1.94/month.\nYou'll need to install the Fly CLI tool:\ncurl -L https://fly.io/install.sh | sh\n\nThen run the following to authenticate with Fly:\nflyctl auth login\n\nYou should also install the datasette-publish-fly plugin:\npip install datasette-publish-fly\n\nWith all of those pieces in place, you can deploy the application like this:\ndatasette publish fly timezones.db \\\n  --app timezones-api \\\n  --setting default_page_size 10 \\\n  --install datasette-geojson-map \\\n  --metadata metadata.yml \\\n  --spatialite\n\nYou need a unique --app name (I've already claimed timezones-api for this demo).\nThe --spatialite flag ensures SpatiaLite is configured for the deployed application.\nThe deploy command make take a few minutes to complete. Once it has finished you can visit https://your-app-name.fly.dev/ to see the finished application, live on the internet.\nHere's timezones.datasette.io, deployed using this exact command.\nSource code\nThe source code for everything in this tutorial can be found in the simonw/timezones-api repository on GitHub."}], "truncated": false}