One thing we could do is iterate through all of the cross-references for the entry then add lots more terms, but with the alternate db and db_id (while keeping the source?) then there would need to be some post-processing logic on all of the terms to remove redundancies

Maybe something like this?

def _generate_obo_terms(prefix, keep_xrefs=None):
    filename = os.path.join(resources, '{prefix}.json'.format(prefix=prefix))
    logger.info('Loading %s', filename)
    with open(filename) as file:
        entries = json.load(file)

    db = prefix.upper()
    if keep_xrefs is None:
        keep_xrefs = set()
    for entry in entries:
        id_ = entry['id']
        name = entry['name']
        yield Term(normalize(name), name, db, id_, name, 'name', prefix)
        for synonym in entry['synonyms']:
            yield Term(normalize(synonym), synonym, db, id_, name,
                       'synonym', prefix)
        curies = [
            (entry['id'], entry['name'], prefix),
        ]

        for xref_db, xref_id in _get_xrefs(entry, keep_xrefs):
            curies.append((xref_db, xref_id, prefix))

        for db_id, name, source in curies:
            yield Term(normalize(name), name, db, db_id, name, 'name', source)
            for synonym in set(entry['synonyms']):
                yield Term(normalize(synonym), synonym, db, db_id, name,
                           'synonym', source)


def _get_xrefs(entry, keep_xrefs):
    for xref in entry['xrefs']:
        try:
            db, db_id = xref.split(':', 1)
        except ValueError:
            continue
        if db in keep_xrefs:
            yield db, db_id

Okay now I'm checking out what the consequences of adding this loop for the xrefs - it makes it slow down from < 1 second to mode than 30 minutes for each. Investigating now...

Lookup of MeSH identifiers solved with gyorilab/indra#933

Remaining missing DOID xrefs seem to be due to alternate identifiers. Looking into that now

Alright, thanks again for this @cthoyt, I finally had a chance to work through this and merge it. I made the following changes:

• Migrated the code from the extra_terms file into the main generate_terms file.
• Reimplemented the xref mapping logic to apply once to a single entry (instead of generating out all xref combinations for each synonym) and replace it with a MESH or DOID ID. I reviewed the types of all the xrefs reported by all EFO, HP and DOID, and from what I can tell, only these two mappings are relevant in this context. I also got rid of mappings to supplementary MESH IDs which were often to outdated entries - in these cases we stick with the original source instead of MESH.
• Reviewed the entries we get from each source and found a number of corner cases that need to be post-processed or removed. There are still some weird synonyms but I handled most of the low-hanging-fruit issues.

As a follow-up, not unexpectedly, there are redundancies not resolved by the source-provided mappings. For instance:

In [11]: ground('pancreatic cancer')                                                                                                          
Out[11]: 
[ScoredMatch(Term(pancreatic cancer,pancreatic cancer,DOID,DOID:1793,pancreatic cancer,name,efo),0.7777777777777778,Match(query=pancreatic cancer,ref=pancreatic cancer,exact=True,space_mismatch=False,dash_mismatches={},cap_combos=[])),
 ScoredMatch(Term(pancreatic cancer,pancreatic cancer,EFO,0002618,pancreatic carcinoma,synonym,efo),0.5555555555555556,Match(query=pancreatic cancer,ref=pancreatic cancer,exact=True,space_mismatch=False,dash_mismatches={},cap_combos=[])),
 ScoredMatch(Term(pancreatic cancer,pancreatic cancer,MESH,D010190,Pancreatic Neoplasms,synonym,efo),0.5555555555555556,Match(query=pancreatic cancer,ref=pancreatic cancer,exact=True,space_mismatch=False,dash_mismatches={},cap_combos=[]))]

In a separate PR, I will attempt to extend the automated MeSH mappings finder concept to these new redundancies.