Redstone GIS

Project archive · The Wilderness Society · 2015–2019

Putting a century of federal leases on the map

The Bureau of Land Management records every oil, gas, coal, wind and solar authorization on public land. For years it published them only as text reports with legal land descriptions. This project scraped those reports with Python and Selenium and turned each description into a polygon.

The decoder on this page does the core step by hand. Type a legal description and it shades the part of a one-mile section it describes.

Python · Selenium · ArcPy 548,745 case records scraped Presented at Esri Developer Summit 2017
Nominal area
0 acres of 640
First division ID
Second division IDs matched

Examples are rows from a California solar right-of-way used to test the tool: San Bernardino Meridian, T3S R15E, Riverside County. IDs follow the PLSS geodatabase key the scripts build.

The problem

You can't plan conservation if you can't see how the land is being used

640 millionacres owned by the federal government, about 28% of the United States
92%of those acres sit in 12 Western states
7kinds of authorization tracked: oil and gas, coal, geothermal, wind, solar, rights-of-way, other minerals

The BLM manages the federal mineral estate, including minerals under land run by the Forest Service, Park Service and Fish and Wildlife Service. Its system of record was LR2000, the Legacy Rehost System. LR2000 could report on any lease or right-of-way, but only as a table.

GIS coverage was patchy. State offices published some layers, not every state published mineral leasing, and none carried everything in LR2000. The BLM's GeoCommunicator site filled part of the gap until its funding was removed in 2011 over concerns about data quality. The boundaries it derived from LR2000's legal descriptions were not accurate enough.

Each record describes its location in the Public Land Survey System: meridian, township, range, section, then aliquot parts such as E2NE. A person can find one in ArcMap by selecting PLSS polygons by hand. The Wyoming state office alone had more than 155,000 case records.

Manual selection = mind-numbing madness.Slide 8, Esri Developer Summit 2017
Federal lands by agency. BLM land in yellow. Map by The Wilderness Society.
The source. LR2000's public reporting site.
What you get back. A list of serial numbers with case type, disposition and acres. No geometry.
What it has to match. PLSS second-division polygons. About 1.5 million polygons in the layer shown.

The pipeline

One ArcGIS toolbox, from search form to finished polygon

The final version is seven Python modules, about 2,500 lines, run from an ArcGIS script tool. It went through 28 numbered revisions of the shape-building module between 2015 and 2016.

  1. 01 · INPUT

    Choose what to pull

    Administrative state, case types, date range, disposition, a list or range of serial numbers, or a master search for everything.

  2. 02 · SCRAPE

    Download the case list

    Selenium drives Firefox through the LR2000 criteria form and saves the summary report. A custom profile sends downloads straight to a folder with no dialog.

  3. 03 · SCRAPE

    Fetch details, 60 at a time

    LR2000 accepts 60 serial numbers per detail request. The script loops until the list is done, with randomized pauses and up to three attempts at each click.

  4. 04 · PARSE

    Split each report into tables

    One detail file holds several stacked sub-tables. Regular expressions separate them into lessee, coordinates, relinquished, action and remarks tables.

  5. 05 · MATCH

    Build the shape

    Each coordinate row becomes a PLSS key. ArcPy selects the matching polygons with a SQL IN query, which ran faster than a join, then dissolves them by serial number.

  6. 06 · MANAGE

    Append to the master database

    Each run is appended to a per-state, per-commodity geodatabase and de-duplicated. A master list records which cases have shapes and which have no detail in LR2000.

A run, recorded in March 2017. Firefox opens LR2000 under Selenium's control, downloads the reports, and the ArcMap tool reports each PLSS query as it builds shapes. Three minutes, no sound.
The front end. Thirteen parameters. Only the output geodatabase is required.
Step 4 input. A serial register page for one right-of-way. Lessee, land description and action history are stacked in one sheet.
Step 6 output. One geodatabase per state and commodity: the shapes plus every related table.
Master list. Arizona oil and gas, with the status of every record.

Matching and error checking

When the fine grid has no match, fall back to a coarser one and say so

The same parcel shows up in LR2000 as lot 1, L1 or 1. Ranges such as 1-4 have to be expanded. Columns that hold comma-separated lists are quoted in the CSV, and some values contain quotes themselves. In places the PLSS layer is simply missing.

The tool normalizes what it can. Anything it still can't match at the quarter-quarter level is drawn from the next level up, and the miss is counted. Every finished shape carries the two counts and an accuracy flag, so a map reader can tell an exact boundary from an approximation.

Per-lease accuracy. A shape is correct only when both error counts are zero.
First try

Second division

Quarter-quarter sections and lots, nominally 40 acres. Exact match on the concatenated key.

If unmatched

First division

The whole section, 640 acres. Counted in SUM_ERROR_SECDIV_NUM.

If still unmatched

Township

The 36-section township. Counted in SUM_ERROR_FRSTDIV_NUM.

Coverage · 12 September 2019

87% of authorized cases in the West had shapes

Counts are by BLM administrative state, across all commodities. About one case in five fell back from the second division for at least one row, and 3.4% fell back to the township.

Admin stateRecords scrapedTotal recordsAuthorized scrapedAuthorized totalAuthorized complete2nd division fallback1st division fallback
Total548,745797,851218,754251,24587%19.89%3.41%

Source: project tracking sheet shown in the September 2019 overview. Eastern States had not been scraped. Remaining at that date: 32,491 authorized cases and 249,106 records in all.

What it made possible

Web maps, lease-sale tracking and a West-wide leasing history

The shapes were loaded to PostgreSQL and ArcGIS Server and published as map services with their related tables. By 2019 the work had a name, the Federal Lands Use & Transparency Tool.

Wyoming oil and gas. From the 2017 talk.

109,283 Wyoming leases by status

  • Authorized 15,071
  • Closed 93,904
  • Pending 190
  • Cancelled 116
  • Relinquished 2

Six of every seven were already closed. Mapping them shows where leasing has been, as well as where it is now.

Oil & Gas Leasing web map. Current and historic leases across the West, updated from LR2000.
Wind Energy Tracker. Renewables were where the project started.
Upcoming lease sales. Parcels proposed for auction, by state and month.
Expressions of interest. Industry nominations, tracked as they were filed.
Leasing through time, 1920 to 2015. Oil and gas leases in red over federal public land in green, stepped year by year. Preliminary research animation, 20 seconds.
Idle versus producing, September 2019. 9,676,956 leased acres were producing. 8,532,391 were sitting idle.

Share of land leased to oil and gas since 1920

Because closed leases were mapped too, the full footprint could be measured. Forty percent of federal public land in the Western states has been under lease at some point.

Of federal public land. From 17% in California to 65% in Wyoming.
Of BLM land. 90% in Wyoming, 95% in Montana.
Of the whole state. 51% of Wyoming, 44% of Utah.
Red Desert-to-Hoback. Leases over a mule deer migration corridor, in a story map with the Center for Western Priorities.
Grand Staircase-Escalante. Coal leases with lessee and action history one click away.
Corporate lands. Public land acres under lease, by company.

Timeline

Four years from a test spreadsheet to a West-wide database

  1. PLSS to Shape

    The first toolbox models turn one California solar right-of-way's land description into polygons.

  2. Full automation

    Selenium scraping, parsing and shape building run as one tool. The shape module reaches version 28.

  3. Esri Developer Summit

    Presented as "Using Python and Selenium to Make Geographical Sense of BLM's LR2000."

  4. FLURTT

    548,745 records scraped. Next steps: extract everything, update services weekly, add mining claims and land transfers.