FutureHouse's AI research toolkit

Asimov Press recently published an interesting interview with FutureHouse, a San Francisco non-profit that tries to automate scientific discovery. They are developing several practical AI-powered tools to better work with research literature¹, and three of them seem particularly interesting to me.

PaperQA: research summarizer and ‘semantic linter’

In their paper on PaperQA, a ‘language model agent’, they show it can not only summarize papers (better than Wikipedia; more on that below) but can also find contradictions in papers. They tested this with randomly sampled biology papers and early results are promising (n is only 93, though):

PaperQA2 identifies 2.34 +/- 1.99 contradictions per paper in a random subset of biology papers, of which 70% are validated by human experts. These results demonstrate that language model agents are now capable of exceeding domain experts across meaningful tasks on scientific literature.

I could see many practical uses of such a tool besides classic literature search, because I’m sure most of the contradictions the tool found are benign and a result of sloppy writing or otherwise avoidable errors. This would be quite useful when preparing a manuscript, in the same way that programmers rely on linters to identify problematic code in a codebase.²

As PaperQA is open source, you can play around with it yourself.³

Has anyone …?

I’m also curious about how well hasanyone.com will work⁴, as this might be a much nicer interface for literature search than Google scholar or using ‘vanilla ChatGPT’. The idea is straightforward: just ask in plain text if there is scholarly work on a specific problem, crucially one that you can describe in your own words⁵. If I understand this correctly, the service then uses an LLM to map your question onto similar concepts across the various scientific subdomains that may have dealt with it, and the result is a brief summary and a list of references. This could be a boon for finding related work and could help to avoid re-inventing the wheel.⁶

WikiCrow as LLM-generated bottom-up OMIM?

Another cool tool that apparently came about by applying PaperQA to Wikipedia is WikiCrow, which provides a Wikipedia-style page for all⁷ human genes, including a nice description and further literature.

I’m not a clinical geneticist but I know that resources like Genomenon’s Mastermind database or OMIM (Online Mendelian Inheritance in Man)⁸ are crucial for diagnostics, and having more options here is always a good thing. When comparing the results of WikiCrow for a well-known gene like BRCA2, you can see that its results are ‘kind of OK’, but other sources still have more and better information:

• Mastermind shows almost 60.000 papers and provides variants as well as relations to diseases and gene-specific metrics that are relevant for a diagnosis (such as gnomAD’s LOEUF).

• OMIM provides much more details on the different aspects to consider (gene structure, phenotype correlations, and so on), as well as ca. 100 curated paper references.

• WikiCrow shows 15 references (this might be limited by the LLMs output size limitations), fails to show BRCA1 as a ‘related gene’, and has some other peculiarities: genomic coordinates are given only for hg19 (the current reference genome is hg38, released end of 2013), the gene’s strand is given as ‘1’ (customary notation would be ‘+’ and ‘-‘, or ‘forward’ and ‘reverse’), and genomic coordinates as well as the number of exons are transcript-specific, and BRCA2 has several transcripts⁹ with differing exon count (so it does not make sense to state that BRCA2 has 26 exons).

In summary: this is a really cool idea, but it’s not quite there yet. I wonder how far the FutureHouse team can push this, though, because my nitpicks are totally solvable problems and I would really like to see how good a service that augments the basic bioinformatics data (for which you don’t need an LLM) with LLM capabilities can become.

–