We are a small team that builds free web tools. One of them is an index of the APIs that national government open-data portals publish, one country per edition: Korea, Japan, the United States, France, Germany, the United Kingdom, Taiwan, Canada, Australia and Singapore. Every week a Python script re-reads each portal and keeps only the entries that can actually be called as an API or a web service.

The interesting part was not the index. It was finding out that ten portals answer the question "which of your entries are APIs?" in ten different ways. These are the notes from building that crawler, with the endpoints we use.

Three families of portal

Roughly, the portals fall into three groups:

• CKAN instances (Germany, the UK, Canada, Australia). You get a generic dataset search and have to work out for yourself which datasets are services.
• Portals with a dedicated API list (France, Korea, Singapore, and the US in its own way). Somebody already decided what counts as an API.
• No list endpoint at all (Japan), or one big export file (Taiwan).

Each family needs a different strategy, and each has one trap that cost us a run.

CKAN: ask the facets first

On a CKAN portal there is no "is this an API?" field. What you have is the format string on each resource, typed by whoever published it. So the first call is not a search, it is a facet count over every format the portal has ever seen:

GET https://data.gov.au/data/api/3/action/package_search?rows=0&facet.field=["res_format"]&facet.limit=-1

On October 18, 2026 that returned 416 distinct format strings on data.gov.au. Twenty-three of them look like a service, and six of those mention WMS alone:

WMS
OGC WMS
wms & wfs
ArcGIS REST WMS WFS
ArcGIS REST  WMS  WFS
arcgis rest wms wfs

So you cannot filter on res_format:WMS and call it done. We match the facet values against one regular expression and then build a single filter query from the exact strings that matched:

SVC = re.compile(
    r"(?<![a-z])(api|rest|wms|wfs|wcs|wmts|sparql|oaf|odata|_srvc)(?![a-z])"
    r"|arcgis.*(server|service)", re.I)

fac = get(f"{base}/package_search?rows=0&facet.field=%5B%22res_format%22%5D&facet.limit=-1")
fmts = sorted(f for f in fac["result"]["facets"]["res_format"] if SVC.search(f.rsplit("/", 1)[-1]))
fq = "res_format:(" + " OR ".join('"%s"' % f.replace('"', '\\"') for f in fmts) + ")"

Two details matter. The lookarounds stop api matching inside words like rapid, and the rsplit("/") handles portals where the format is a URI (some EU-style vocabularies) rather than a word. The arcgis.*(server|service) branch keeps ArcGIS REST services while leaving out ArcGIS Online story maps and web apps, which are pages, not APIs.

Then page through with a stable sort:

r = get(f"{base}/package_search?" + urlencode(
    {"fq": fq, "rows": 1000, "start": start, "sort": "id asc"}))["result"]

sort=id asc is there on purpose. Paging with the default relevance sort is not guaranteed to be stable between requests, and on a catalog that is being harvested while you read it, rows can move between pages.

The base URLs differ more than you would expect for the same software: ckan.govdata.de/api/3/action for Germany, ckan.publishing.service.gov.uk/api/action for the UK (we switched to it because data.gov.uk/api returned an error page on October 11, 2026), open.canada.ca/data/api/action for Canada and data.gov.au/data/api/3/action for Australia. Canada also has an API collection, so we run a second filter, collection:api, and merge the two result sets by dataset id.

Portals that already list their APIs

France. data.gouv.fr has a separate API catalog, /api/1/dataservices/, distinct from its datasets. Everything in it is an API, so the only filtering is dropping entries with archived_at or deleted_at set. It pages with a next_page link, and each record carries base_api_url and a machine-readable documentation URL when the publisher filled one in. On October 18 the endpoint reported 1,266 entries.

Korea. data.go.kr publishes its own list of open APIs as an API (dataset 15077093, served from api.odcloud.kr), which needs a free service key. The list is already API-only. Licence and the type of the providing body are not in that list, so we read them from each API's catalog JSON and from the portal search page.

Singapore. The v2 public endpoint api-production.data.gov.sg/v2/public/api/datasets?page=N returns every dataset with a format field. We keep format == "API" and status == "active". The portal's CSV and GeoJSON datasets are queryable through the same API, but they are file datasets, so they stay out of the index.

United States. Data.gov's catalog search has no format filter. We run four searches (API, "OGC WMS", WMS, WFS) and keep only records whose distributions actually carry one of those types, de-duplicating by identifier. This is where paging bit us: paging the relevance-sorted results with the after cursor returned overlapping rows. On October 11 one run collected 25,842 rows of which only 20,560 were unique, and it stopped about halfway. Sorting by last_harvested_date removed the overlap. The search also returns HTTP 429 if you go too fast, so there is a two-second pause between pages.

No list endpoint: read the pages

Japan. The Digital Agency's e-Gov API Catalog has no endpoint that returns the list. We read the list page 50 at a time until the count printed on the page (「件のAPIが見つかりました」) is reached, then open each detail page and pull fields by their labels: providing organisation, API format, response format, public URL. It is 38 entries, so a one-second delay per request is cheap. The e-Gov data portal had zero resources registered in API format on October 11, so it is not used.

Taiwan. data.gov.tw offers the whole catalog as a single JSON export. We keep datasets whose provision type is 「API服務」 or whose file formats include WEBSERVICES, API, WMS or WFS. Of 52,513 datasets in the October 17 export, 1,045 qualified.

What a weekly run looks like

For CKAN and the US, "total in source" is already the filtered search result, which is why the two columns match. For Korea, the first number is the count on the portal's search page and the second is what the list API returned, which was slightly higher.

One guard worth copying

Portals fail in quiet ways: an error page with HTTP 200, a search that returns half its usual rows, an expired key. The crawler compares each country's new row count with its last good run and refuses to overwrite the data if it shrank by more than half, exiting with code 1. The weekly job calls each country separately, so one broken portal keeps last week's edition while the other nine update.

If you only need one portal, the CKAN facet trick and the French dataservices endpoint are the two pieces most worth reusing. If you want to search all ten without writing the crawler, the index is at govapiatlas.com. It does not host the APIs; applying for keys and calling them happens on each portal.

"k-anonymity" gets attached to the Pwned Passwords range API in almost every write-up, usually without a value for k. If you are adding a breached-password check to a signup flow, that number tells you how much the API operator, or anyone watching the network, could infer from the one thing you send.

This post measures it: one short Node script, real responses, and the design decisions that follow.

What the documentation promises

The API documentation describes the contract:

• You send the first five hex characters of a SHA-1 (or NTLM) hash to https://api.pwnedpasswords.com/range/{prefix}.
• You get back every known hash suffix that shares that prefix, one per line, as SUFFIX:COUNT.
• There are 16⁵ = 1,048,576 possible prefixes, and every one returns HTTP 200.
• No API key, and no rate limit.
• An optional Add-Padding: true request header adds decoy rows with a count of 0.
• ?mode=ntlm switches the response to NTLM suffixes.

The same page says a typical range returns approximately 800 suffixes and that padded responses contain between 800 and 1,000 rows. Those two figures are the ones worth checking, because the data set keeps growing.

The script

No dependencies. It needs Node 18 or later for the built-in fetch and AbortSignal.timeout.

// survey.mjs — run with: node survey.mjs
import { createHash } from 'node:crypto';

const sha1 = text => createHash('sha1').update(text, 'utf8').digest('hex').toUpperCase();

async function bucket(prefix, { padding = false, mode } = {}) {
  const url = `https://api.pwnedpasswords.com/range/${prefix}${mode ? `?mode=${mode}` : ''}`;
  const headers = { 'User-Agent': 'range-bucket-survey' };
  if (padding) headers['Add-Padding'] = 'true';
  const res = await fetch(url, { headers, signal: AbortSignal.timeout(5000) });
  if (!res.ok) throw new Error(`HTTP ${res.status} for ${prefix}`);
  const body = await res.text();
  const rows = body.trim().split(/\r?\n/).map(line => {
    const [suffix, count] = line.split(':');
    return { suffix, count: Number(count) };
  });
  return { rows, bytes: Buffer.byteLength(body) };
}

// 1. One lookup, end to end, with a password everyone knows is burned
const hash = sha1('password');
const prefix = hash.slice(0, 5);
const suffix = hash.slice(5);
const plain = await bucket(prefix);
const hit = plain.rows.find(row => row.suffix === suffix);
console.log(`prefix ${prefix}, suffix length ${suffix.length}`);
console.log(`plain : ${plain.rows.length} rows, ${plain.bytes} bytes, count for "password" = ${hit ? hit.count : 0}`);

// 2. The same bucket, padded, twice
for (const run of [1, 2]) {
  const padded = await bucket(prefix, { padding: true });
  const zeros = padded.rows.filter(row => row.count === 0).length;
  console.log(`padded run ${run}: ${padded.rows.length} rows (${zeros} with count 0), ${padded.bytes} bytes`);
}

// 3. How big are other buckets?
const sizes = [];
for (const p of ['00000', '7C4A8', 'ABCDE', 'FFFFF']) {
  const { rows } = await bucket(p);
  sizes.push(rows.length);
  console.log(`bucket ${p}: ${rows.length} rows`);
}
const mean = Math.round(sizes.reduce((a, b) => a + b, 0) / sizes.length);
console.log(`mean of those four: ${mean} rows; x 1,048,576 buckets = about ${(mean * 1048576 / 1e9).toFixed(1)} billion hashes`);

// 4. NTLM mode
const ntlm = await bucket(prefix, { mode: 'ntlm' });
console.log(`ntlm suffix length: ${ntlm.rows[0].suffix.length}`);

What came back

Output from a run on October 9, 2026. Your row counts will differ a little, since the data set changes and padding is random.

prefix 5BAA6, suffix length 35
plain : 1978 rows, 77639 bytes, count for "password" = 52372427
padded run 1: 2020 rows (42 with count 0), 79277 bytes
padded run 2: 2174 rows (196 with count 0), 85283 bytes
bucket 00000: 2509 rows
bucket 7C4A8: 1921 rows
bucket ABCDE: 2046 rows
bucket FFFFF: 2047 rows
mean of those four: 2131 rows; x 1,048,576 buckets = about 2.2 billion hashes
ntlm suffix length: 27

Four observations.

k is around 2,000 now. Every bucket we sampled held between 1,921 and 2,509 suffixes, well above the "approximately 800" in the documentation's description. Five buckets is a tiny sample, so treat 2,000 as an order of magnitude and the 2.2 billion extrapolation as a rough estimate. The direction is plain enough: the anonymity set has grown with the corpus.

The response is not small. About 78 KB of text once decoded. On a signup endpoint that is the dominant cost of the check, larger than the hash and comparison by far. It is worth a timeout, which the script sets at five seconds.

Padding added tens to hundreds of rows. The two padded calls to the same bucket returned 2,020 and 2,174 rows, with 42 and 196 zero-count decoys. Because the unpadded bucket is already past 1,000 rows, the padded size here is "real rows plus a random extra", not the 800 to 1,000 range the documentation describes. The property that matters still holds: two requests for the same prefix produced different lengths, so response size alone is a weaker signal of which bucket was fetched.

NTLM suffixes are 27 characters. An NTLM hash is 32 hex characters, so five go in the URL and 27 come back. That matches the documentation and is useful if you are auditing Windows credential stores, where NTLM is what you already hold.

What k does and does not hide

With five hex characters you reveal 20 bits of a 160-bit hash. The operator learns that the password you are checking is one of roughly 2,000 known-breached candidates in that bucket, or something that is in none of them. For a password that is not in the corpus, which is the outcome you hope for, the prefix narrows nothing useful: the remaining 140 bits are unknown.

What the scheme does not hide is the request itself. The operator and the network path see an IP address, a time and a prefix. If your server makes the call on behalf of users, that IP is yours, not theirs, which is an argument for doing the lookup server-side in a signup flow. If the browser makes the call, the user's IP is exposed to the API and the password never reaches your server for this purpose. Both are defensible. Pick one deliberately.

Decisions for a signup or password-change check

NIST's current digital identity guideline says verifiers "SHALL compare" a prospective password against a blocklist that includes passwords obtained from previous breach corpuses. A range lookup is one way to meet that. A few choices decide whether it behaves well in production.

Fail open or fail closed? If the API times out, you can reject the signup or let it through. Rejecting turns a third party's outage into yours. Accepting quietly means a breached password occasionally slips in. A reasonable middle: accept, record that the check was skipped, and re-run it at next login. Whatever you choose, never report "not found" when you mean "could not check".

Compare the whole suffix, exactly. The response uses uppercase hex. Normalize your own hash to uppercase once and do an exact string match on all 35 characters. With padding on, a decoy row can only be mistaken for a hit if you ignore the count, so treat a matched row with count 0 as no match.

Do not transform the password first. Hash the exact bytes the user typed, in UTF-8. Trimming or lowercasing checks a different password from the one being set.

Cache by prefix if you must, briefly. A bucket is the same for every user who lands in it, so a short-lived cache keyed by prefix is safe from a privacy standpoint and saves the 78 KB download. Keep it short, because new breaches are loaded into the corpus. The documentation notes there are no licensing or attribution requirements on this API, so nothing in the terms forbids holding a response for a few minutes.

Send a User-Agent. The documentation's general rules require one and describe a 403 without it. When we tried the range endpoint with the header removed it still answered 200, but that is behaviour, not contract. Set a descriptive value and you never have to find out which endpoints enforce it.

What this check is not

A clean range lookup says the password is not in one corpus. It says nothing about whether the password is short, guessable, or reused by this user elsewhere. Keep the length rule and any strength estimate as separate checks with separate messages.

We are a small team that builds free web tools, and one of the things we build is a browser-based checker that follows this split: the breach lookup hashes locally and sends five characters of the hash, while a strength score is computed on the device and shown beside it. If you are designing the user-facing side of this, it is a quick way to see the two results presented as two things.

See the lookup and the strength estimate side by side: pwcheck.fyi

Here is a puzzle any QR generator can produce: you change one word in a link and the code comes out visibly denser. Nothing is wrong with the generator. The payload crossed a capacity boundary and the encoder moved up a version, which makes the symbol four modules wider and taller.

Where those boundaries fall depends on something most generator UIs never show: the encoding mode. This post makes it visible with a small script, then turns the result into rules you can apply before a design is locked.

Four modes, four prices

A QR symbol stores a bit stream. The standard defines several ways to turn characters into bits, and they are not equally compact. DENSO WAVE, which created the format, lists the maximum capacity of the largest symbol as 7,089 numeric, 4,296 alphanumeric, 2,953 byte and 1,817 kanji characters. Same symbol, same number of bits; the difference is the price per character.

Two details in that table do most of the damage in practice. The alphanumeric set has no lowercase letters. And byte mode charges per byte, not per character, so the cost of non-Latin text depends on how you serialized it.

Measure it

The script uses the qrcode package's create() function, which returns the symbol's version and the segments the encoder chose. Install it with npm i qrcode@1.5.4.

// modes.js — npm i qrcode@1.5.4 && node modes.js
const QRCode = require('qrcode');
const toSJIS = require('qrcode/helper/to-sjis');

function describe(label, text, options = {}) {
  const qr = QRCode.create(text, { errorCorrectionLevel: 'M', ...options });
  const modes = qr.segments.map(s => `${s.mode.id}(${s.getLength()})`).join(' + ');
  const bytes = Buffer.byteLength(text, 'utf8');
  const size = qr.modules.size;
  console.log(`${label.padEnd(16)} ${String(text.length).padStart(3)} chars ${String(bytes).padStart(3)} bytes  v${qr.version} ${size}x${size}  ${modes}`);
}

const url = 'https://example.com/menu/autumn-2026?table=12';
describe('mixed-case URL', url);
describe('upper-case URL', url.toUpperCase());
describe('digits only', '8801234567890'.repeat(3));
describe('korean', '가을 메뉴판은 여기에서 확인하세요');
describe('japanese', '秋のメニューはこちらからご覧ください');
describe('japanese (kanji)', '秋のメニューはこちらからご覧ください', { toSJISFunc: toSJIS });

console.log('');
for (const level of ['L', 'M', 'Q', 'H']) {
  describe(`URL at level ${level}`, url, { errorCorrectionLevel: level });
}

console.log('');
// How many repeats of one character fit in version 3 at level M?
function fits(unit) {
  let n = 0;
  while (QRCode.create(unit.repeat(n + 1), { errorCorrectionLevel: 'M' }).version <= 3) n++;
  return n;
}
console.log(`v3-M holds: ${fits('7')} digits | ${fits('A')} upper-case | ${fits('a')} lower-case | ${fits('가')} hangul`);

Output:

mixed-case URL    45 chars  45 bytes  v4 33x33  Byte(45)
upper-case URL    45 chars  45 bytes  v3 29x29  Alphanumeric(36) + Byte(9)
digits only       39 chars  39 bytes  v2 25x25  Numeric(39)
korean            18 chars  48 bytes  v4 33x33  Byte(48)
japanese          18 chars  54 bytes  v4 33x33  Byte(54)
japanese (kanji)  18 chars  54 bytes  v3 29x29  Kanji(18)

URL at level L    45 chars  45 bytes  v3 29x29  Byte(45)
URL at level M    45 chars  45 bytes  v4 33x33  Byte(45)
URL at level Q    45 chars  45 bytes  v4 33x33  Byte(45)
URL at level H    45 chars  45 bytes  v6 41x41  Byte(45)

v3-M holds: 101 digits | 61 upper-case | 42 lower-case | 14 hangul

Reading the output

One lowercase letter forces byte mode. The mixed-case URL is a single 45-byte segment at version 4. Uppercased, the same address splits into an alphanumeric run of 36 characters and a 9-byte tail, and fits version 3. The tail is ?TABLE=12: neither ? nor = is in the alphanumeric set, so the encoder falls back to bytes from the first character it cannot represent.

Digits are almost free. Thirty-nine digits fit in version 2. A numeric ticket ID or a GTIN is the cheapest thing you can encode.

CJK text costs three bytes a character in UTF-8. Eighteen Korean characters, three of them spaces, serialize to 48 bytes. Eighteen Japanese characters come to 54. Both need version 4, the same as a 45-character Latin URL. The last line of the output puts a number on it: version 3 at level M holds 42 lowercase letters or 14 Hangul syllables.

Kanji mode claws that back for Japanese. With a Shift JIS conversion function supplied, the same Japanese sentence encodes as 18 kanji-mode characters and drops to version 3. There is no equivalent compact mode for Hangul, so Korean text always pays the UTF-8 price.

Error correction level is the other multiplier. The 45-character URL is version 3 at L, version 4 at M and Q, and version 6 at H. Going from M to H added eight modules to each side. If a design adds a centre logo and the tool raises the level to H to compensate, the symbol grows even though the payload did not change.

Can you just uppercase the URL?

Partly. RFC 3986 says the scheme and host are case-insensitive, and that the other components are assumed to be case-sensitive unless the scheme says otherwise. So HTTPS://EXAMPLE.COM is equivalent to its lowercase form, and whether /MENU/AUTUMN-2026 reaches the same page as /menu/autumn-2026 depends on your server.

If you control the server, a route that is deliberately case-insensitive and avoids ?, = and & can stay entirely in alphanumeric mode. HTTPS://EXAMPLE.COM/M/12 is 24 characters at 5.5 bits each. That is a legitimate optimization for codes printed very small. For everything else, the simpler win is a shorter path.

The single-byte trap

Byte mode stores bytes. Some encoder libraries, including the rendering library our own generator uses, treat each character of a JavaScript string as a single byte, which corrupts anything outside Latin-1 unless you prepare the string first. The usual fix is to hand the library a string in which each character already stands for one UTF-8 byte:

const utf8 = s => unescape(encodeURIComponent(s));

const s = '가A';
console.log([...utf8(s)].map(c => c.charCodeAt(0).toString(16)).join(' '), '| length', s.length, '→', utf8(s).length);
// ea b0 80 41 | length 2 → 4

encodeURIComponent percent-encodes the UTF-8 bytes, and unescape turns each %XX back into a single character with that code. The result is a "binary string" four characters long whose character codes are the four UTF-8 bytes. unescape is a legacy function; new TextEncoder().encode(s) gives you the same bytes as a Uint8Array if your library accepts one. Either way, check a non-Latin payload by scanning it, and budget by Buffer.byteLength(text, 'utf8') or TextEncoder, never by text.length.

A budget you can apply before design starts

Using version 3 at level M (29 × 29 modules) as the target for a small print:

Rules that follow:

1. Budget in bytes for anything that is not purely digits or the alphanumeric set.

2. Keep query strings out of printed codes where you can. ? and = end an alphanumeric run, and tracking parameters are usually the longest part of the payload.

3. Put non-Latin text behind a short URL instead of into the symbol, unless it is Japanese and your encoder and target scanners handle kanji mode.

4. Decide the error correction level before measuring, and measure again if a logo is added later.

5. Log the version in your build or export step. A jump from one version to the next is the earliest warning that a payload change will alter the artwork.

Where this shows up in a generator

We are a small team that builds free web tools, and one of the things we build is a QR code generator that runs entirely in the browser. It applies the UTF-8 conversion shown above before handing text to its rendering library, so Korean and Japanese payloads scan correctly, and it switches error correction to H when a logo is added. Both behaviours are the reason a code can change density when the visible text barely changed, which is where this post started.

The script is the more useful takeaway. Point it at your own payloads and you will know the version, and so the module count, before anyone opens a design tool.

Try payloads of different kinds and watch the symbol change: makeqr.fyi