PubMed search and topic clustering

GL1X searches PubMed, the U.S. National Library of Medicine's index of the biomedical literature, and shows what a search finds as a map of its topics. Up to 2,000 records come from NCBI and are grouped in the browser by Broccoli, Decentralized Science Labs' clustering library, with Lingo, Suffix Tree Clustering (STC) or bisecting k-means. Click a topic and the map zooms into it while its records are clustered again, so a literature can be read from its broad themes down to single papers.

Every structure in the explorer links here: by its PDB ID, and by its title, which the explorer first digests into the search that finds the structure's literature, next to its links to the research workbench and to its record.

Where: the PubMed search page (/bio/pubmed, in the left menu under Bioinformatics onchain → Search), the PubMed links of every structure, the API (/api/bio/pubmed/…) and MCP (pubmed_search).

Searching #

The search box takes PubMed's own syntax, as on pubmed.ncbi.nlm.nih.gov: plain words (PubMed maps them to MeSH terms, journals and authors), field tags such as [tiab] (title and abstract), [mh] (MeSH heading), [au] (author) or [dp] (publication date), quoted phrases, and the operators AND, OR and NOT in capitals.

When a search finds nothing, the page offers NCBI's spelling suggestion (ESpell). The records are fetched in batches of 200, and a progress bar shows each stage.

Settings:

• Search filters: years from and to, only records with an abstract, English only, humans only, free full text only, reviews only, clinical trials only. The filters are added to the query in PubMed's syntax (hasabstract, english[lang], humans[mh], free full text[sb], review[pt], clinical trial[pt]), the years as its publication dates.
• Text: cluster MeSH headings and author keywords with the titles and abstracts; ignore scientific boilerplate (words like "results suggest" and labels like "Control Group"); your own stop words and stop labels, one a line, * as a wildcard; the label style (Title Case, Sentence case or as written).
• Parameters: every parameter of the chosen algorithm, with a reset to the defaults.

Defaults. The explorer's operator chooses what the page starts with — the clustering algorithm, the map and the records a search fetches — on the admin's Site tab (Admin); the explorer's own are Lingo, the treemap and 200 records. What a visitor changes is kept in their browser, and only that: whatever they leave alone follows the operator's defaults, even after the operator changes them. A link's choices hold for the view it opens and never become the visitor's; a link that names no algorithm, map or number of records (the structures' links name only their search) opens with this browser's own choices, or the operator's defaults where it has none.

The map of topics #

Four maps show the same topics, switchable at any time; a button shows the map on the whole screen.

• Treemap (the default): topics as organic cells, each sized by its records, the subtopics as quieter cells inside them. A topic with no subtopics shows dots for its records; a group too small to split shows its records as a honeycomb, each cell a record with its title.
• Bubbles: topics as circles sized by their records; the beads along a bubble's edge are its subtopics.
• Radial: topics in a ring around the current group, subtopics in the rings outside.
• Florets: topics as the florets of a broccoli head on stems from the query, one bud per record.

Click a topic, on the map or in the list beside it, to zoom in: its records are clustered into subtopics, as deep as they allow; a group too small to split shows its records one by one. On the treemap the subtopics drawn inside the cell grow into the new map, keeping their places. Zoom out with the path above the map, the Zoom out button, a click between the bubbles, the Escape key or the browser's back button. On the treemap, the bubbles and the radial map the arrow keys move between topics and Enter zooms in; on the florets, Tab moves between them. A background worker clusters the topics you are likely to open next, so most zooms are instant.

Below the map, the records of the topic you are in: a citation, the words of the topic's label highlighted, the abstract with its sections, MeSH terms, links to PubMed, PubMed Central and the DOI, and flags for retractions and notable publication types. They can be exported as RIS (Zotero, EndNote, Mendeley), BibTeX or CSV (with each record's topics), or as JSON (the query, the records and every level of topics explored); the PMIDs can be copied, or the records opened in PubMed.

The address holds the query, the records and algorithm, the order and years when they are not the defaults, the filters, the map style when it is not the default, and the topic you zoomed into (/bio/pubmed#q=lysozyme&n=200&a=stc&f=abstracts&v=radial&p=…), so a view can be bookmarked or shared and finds the same records and topics anywhere, whatever the defaults. A link that names only the query (as the structures' links do) is searched with this browser's records, algorithm, filters and map style. The part after # stays in the browser: search engines do not crawl it, and it never reaches the server.

How the maps are drawn #

The treemap is a Voronoi treemap (Balzer and Deussen, 2005). Each topic is a cell of a weighted power diagram (Aurenhammer, 1987) clipped to the map: every point belongs to the topic whose site is nearest once each site's weight is taken off, so every cell is a convex polygon. The weights are solved by Newton's method, the change of a cell's area with the weights being its shared edges' lengths over twice the distance between the sites, until every cell's area is its topic's share of the records to 0.2%; the sites then move towards their cells' centroids (Lloyd, 1982) five times, the weights solved again each time, which rounds the cells. Subtopics are a diagram of their own inside their topic's cell, and zooming in starts the next map from where they were drawn. Shapes morph from one level to the next: every cell is kept as 48 points along its outline, its corners among them, and each point moves to its place in the new map. Should a diagram not resolve, the map falls back on squarified rectangles (Bruls, Huizing and van Wijk, 2000).

The colours are eight matte hues in a fixed order — blue, apricot, teal, lavender, sage, rose, sky, sand — computed in OKLCH (Ottosson, 2020) for the explorer's white and near-black surfaces. Each hue has a soft fill for cells and bubbles, a lighter top for their gradient, an outline, a dark ink of the same hue for the names on the fill (at least 6.9:1 on the eight fills and 5:1 on every other fill, variant and gradient, on either theme) and a stronger mark for beads, buds and swatches, whose order passes the colour-blind (Machado, Oliveira and Fernandes, 2009) and contrast checks against the surface. Every topic is named on the map, so colour only tells neighbours apart. Subtopics wear quiet variants of their topic's fill; Other topics is a cool grey with a hatch.

How the topics are found #

• Lingo (Osiński, Stefanowski and Weiss, 2004) builds a term–document matrix of the titles and abstracts (log tf-idf, title words weighted up) and factorizes it into concepts (non-negative matrix factorization with the page's preset for abstracts, a partial singular value decomposition otherwise). Each concept is named by the best of the records' frequent phrases; a topic holds every record that contains its label, and topics that share most of their records are merged. When the chosen phrase barely beats a shorter one inside it that covers at least twice as many records, the shorter one is kept ("Breast Cancer" rather than "Breast Cancer Screening").
• STC (Zamir and Etzioni, 1998) finds the phrases records share with a suffix array (in place of the suffix tree; the result is the same). Each phrase is a base cluster scored by its length and its records; base clusters that overlap strongly are merged.
• Bisecting k-means (Steinbach, Karypis and Kumar, 2000) runs spherical k-means on tf-idf vectors and keeps the best of several trial bisections at each split.

Words are stemmed (Porter2 for English) and filtered by stop lists, with biomedical additions; terms like IL-6, COVID-19, CD4+ and TNF-α stay whole. The clustering runs in a Web Worker on your own device: the explorer's server only relays the searches to NCBI.

From a structure to its literature #

Every structure the explorer lists (the hits of the molecular search, BLAST, structure and motif search, the archive searches of pockets, interfaces, patterns, templates and modifications, a protein's structures, a ligand's binding sites, the workbench's neighbours, alignments and clusters, the records of a collection) carries the same links, beside the ones the listing already had:

The record page has the same two searches beside its other actions (PubMed by title, PubMed 1AKI), and so does the workbench's header. A title that is only the ID gives one link.

How a title becomes a search #

A structure's title is rarely a good PubMed query as it stands. "AlphaFold monomer v2.0 prediction for putative cell agglutination protein PFL3 (Q874R4)" finds nothing, because PubMed requires every word; "putative cell agglutination protein" finds 160 records. So the PubMed link carries the whole title (/bio/pubmed#title=…), and the explorer digests it (GET /api/bio/pubmed/digest):

1. What never helps a search goes: the prediction prefix ("AlphaFold monomer v2.0 prediction for", "ESMFold model of"), the method ("crystal structure of", "solution NMR structure of", "structural studies of", "the orthorhombic form of"), the resolution ("at 1.60 Å resolution"), accessions and PDB IDs, asides in parentheses that are an identifier or an abbreviation, PubMed's syntax characters and the stop words. A title in capitals is read in lower case.
2. The title is read as a structure's title is written: the subject, what it is bound to ("in complex with", "complexed with", "bound to", "with", "glycine-bound"), and where it comes from ("from Bacillus circulans", "human", "Pseudomonas aeruginosa"). A subject made only of ligand words ("a potent small molecule inhibitor bound to cathepsin K") swaps with its partner.
3. Every word gets a kind: the subject's words; names with digits (RAG1, HIV-1, T4); identifiers (locus tags such as TM1621, compound codes such as 18F or KNI-272, a gene name after "protein" such as PFL3); mutations (D151N, "mutant"); structural and descriptive words ("domain", "complex", "protein", "putative", "precursor"); the partner; the organism.
4. Stages, each leaving out one more kind: all the important words; without identifiers and mutations; without structural and descriptive words; without the partner; without the organism; its three, then two most telling words; without names with digits. A stage that names nothing ("protease" alone, "dimerization domain") is never made.
5. PubMed is asked how many records each stage finds, in order (ESearch with no records, cached like every search), and the first that finds 10 or more is chosen. When none does, the one that finds the most; when none finds any, the important words. A stage of one short word ("dest") finds every record that spells it, so it is asked only when nothing narrower found a record. When NCBI stops answering first (busy or slow), the digest gives the best search it knows — the one that found the most, or, when those asked found nothing, the next one not yet asked — marked incomplete, and no cache keeps it.

The counts are PubMed's in October 2026. The page then runs the chosen search (the address becomes #q=…) and says above the map which words of the title it searched for, with the other stages as buttons to search for instead (with their counts, where PubMed was asked); About this result lists them too. A title that names nothing to search for (only identifiers and structural words, "Uncharacterized protein (A0A0B4J2F0)") is not searched: the page says so and leaves its words in the search box to edit. When the explorer cannot digest the title (the request fails), the page searches the title's words without its method and resolution, and says so.

How the explorer reaches NCBI #

The page does not call NCBI itself: it sends its E-utilities requests to the explorer, which forwards them.

• Only PubMed: ESearch, EFetch, ESpell, ESummary and EInfo on db=pubmed, by GET or by POST with a form of at most 256 KB.
• Not for other sites: a request a browser makes from another site's page is refused (Sec-Fetch-Site, or an Origin that is not this host), so no other site can spend this explorer's NCBI allowance through its visitors. Programs that send neither header are served like any API client, within the budget per address.
• NCBI's rules: the explorer adds its NCBI API key, when the operator has set one, and identifies itself (tool=gl1x and the operator's contact e-mail), so the key never reaches a browser. It stays within NCBI's rate for the whole site, 3 requests a second without a key and 9 with one (NCBI allows 10), shared by the web processes; a request that would wait more than 10 seconds for its turn is answered at once with 429 and Retry-After. The pass-through relays NCBI's own answers, its 429 and server errors included, and the page retries them (up to four times); the API routes retry them twice themselves.
• A cache: NCBI's answers are kept in memory (48 MB): searches, counts and spelling suggestions 30 minutes, records 24 hours. Answers are passed on as NCBI gives them.
• A budget per address: 40 requests at once, then two a second.

A title's digest asks NCBI one count a stage, mostly one to three, at most eight, within 20 seconds; it stops early when NCBI is busy and gives what it knows.

The explorer keeps no record of what is searched. The nginx configuration it installs leaves /api/bio/pubmed/ out of its access log and out of its cache. GL1X_PUBMED=off turns the PubMed search off: the routes answer 503, agents are not offered pubmed_search, and the interface leaves out the menu entry and the PubMed links (the page says the search is off); see Operations.

API and MCP #

An article has pmid, type (article or book), title, vernacular_title, abstract, abstract_sections (label, category, text), authors (name, last_name, fore_name, initials, collective, orcid, affiliations), author_string, journal (title, iso_abbreviation, issn, volume, issue, pages), year, pub_date, doi, pmcid, mesh (descriptor, ui, major, qualifiers), keywords, publication_types, languages, chemicals, status and url (its PubMed page).

Budget: 10 searches, digests or lookups, then one every 3 seconds per address. Errors: missing_query, missing_title, query_and_title (give one), title_too_long (400); pubmed_busy (503, many requests waiting for NCBI; retry after the Retry-After seconds), pubmed_timeout (504), pubmed_unavailable (502), pubmed_off (503).

The MCP tool pubmed_search takes query, title (a structure's title, digested; the answer's digest shows every search tried with its count) or pmids, and the same options (max, start, sort, mindate, maxdate); see MCP. An agent goes from a structure to its literature in two calls: get_molecule gives the title and the PDB ID, pubmed_search with title the papers.

curl -s 'https://explorer.example/api/bio/pubmed/search?q=lysozyme+crystal+structure&max=5'
curl -s 'https://explorer.example/api/bio/pubmed/digest?title=Crystal+structure+of+human+DHODH+with+18F'

Limits #

• 2,000 records a search on the page (the clustering is meant for result sets, not for all of PubMed); 100 a request through the API, from the first 10,000 of a search.
• Records without an abstract have only their title to be clustered on; Only records with an abstract leaves them out.
• The clustering is lexical: topics come from the words records share, not from what the words mean. The same records with the same settings always give the same topics (the algorithms' random choices are seeded), and the same topics the same map.
• A title's digest reads words, not meaning: it knows the usual shapes of PDB and model titles, not every protein's name, and a name with digits it takes for an identifier (a gene name after "protein") is left out only when the search without it finds more. The other stages are a click away.
• PubMed's abstracts may be protected by their publishers' copyright; the explorer shows them as PubMed provides them, with their links.

References #

• Osiński S, Stefanowski J, Weiss D (2004) Lingo: search results clustering algorithm based on singular value decomposition. In: Intelligent Information Processing and Web Mining (IIS 2004), Springer, pp 359–368.
• Osiński S (2006) Improving quality of search results clustering with approximate matrix factorisations. In: Advances in Information Retrieval (ECIR 2006), Springer, pp 167–178.
• Zamir O, Etzioni O (1998) Web document clustering: a feasibility demonstration. In: Proceedings of SIGIR '98, ACM, pp 46–54.
• Steinbach M, Karypis G, Kumar V (2000) A comparison of document clustering techniques. KDD Workshop on Text Mining.
• Lee DD, Seung HS (2001) Algorithms for non-negative matrix factorization. Advances in Neural Information Processing Systems 13:556–562.
• Balzer M, Deussen O (2005) Voronoi treemaps. In: IEEE Symposium on Information Visualization (InfoVis 2005), pp 49–56.
• Aurenhammer F (1987) Power diagrams: properties, algorithms and applications. SIAM Journal on Computing 16(1):78–96.
• Lloyd SP (1982) Least squares quantization in PCM. IEEE Transactions on Information Theory 28(2):129–137.
• Bruls M, Huizing K, van Wijk JJ (2000) Squarified treemaps. In: Data Visualization 2000 (VisSym), Springer, pp 33–42.
• Ottosson B (2020) A perceptual color space for image processing (OKLab). https://bottosson.github.io/posts/oklab/
• Machado GM, Oliveira MM, Fernandes LAF (2009) A physiologically-based model for simulation of color vision deficiency. IEEE Transactions on Visualization and Computer Graphics 15(6):1291–1298.
• Sayers E (2010–) A general introduction to the E-utilities. In: Entrez Programming Utilities Help. National Center for Biotechnology Information. https://www.ncbi.nlm.nih.gov/books/NBK25497/