Web Service Interfaces to LSDBs - Browser Point of View

This post discusses web service interfaces to Locus Specific Databases (LSDBs), from the specific point of view of visualising the data. The aim is to make our experiences of this at NGRL clear and to suggest, on this basis, why web services are desirable and in the broadest sense what they should do to satisfy the "variant browser" use case.

Introduction

For visualisation of LSDB data, one of the major advantages of web service interfaces to LSDBs would be dynamic retrieval of variant data. These would ensure that the data could always remain up-to-date in the browser. In the NGRL Universal Browser, for instance, local copies are made for all external variant databases. This is simply because it is very rare to find a machine-to-machine interface to access such data. The only database that is accessed live is NGRL's own Diagnostic Mutation Database (DMuDB). Live access provides significant advantages in maintainability, particularly with regards to ensuring data is kept up-to-date and avoiding complex/long-winded import procedures.

Standard web service interfaces would also allow users to more easily visualise LSDB data without requiring that the browser software have any prior knowledge of the LSDB in question. With appropriate service discovery mechanisms and standardised interfaces any compliant LSDB could essentially be plugged in to any compliant visualisation tool. The standardised interfaces and exchange formats that would come with web services would also help avoid the potentially arduous task that currently presents itself to the developers of visualisation tools when seeking to integrate new LSDBs.

Problems

The problems include the following:

• Heterogeneous Data Schema: Different LSDBs will have different fields, name them differently, etc. A separate parser is required for each different schema.
• Heterogeneous Data Formats: Even if the data is the same the file formats can differ (e.g. HTML, XML, TSV, CSV, etc.). Separate code to read the file is needed for each different file type.
• Heterogeneous Transport Protocols: How one gets hold of the data varies between LSDBs e.g. HTTP (GET versus POST), FTP, SOAP. Separate code is potentially needed to retrieve data using each different transport protocol.
• Lack of Machine Interpretability: The amount of structure present in LSDBs varies. Generally, the greater reliance on natural language, the harder it is to process data automatically. Different terminology for the same concept is also common. Each different terminology requires more code.
• Heterogeneity in Nomenclature: HGVS nomenclature is by no means universally used. Where it is used it is often incorrect. For each different nomenclature and each different misuse of the nomenclature special cases have to be coded.
• Heterogeneity in Reference Sequences: In the worst case LSDBs do not state a reference sequence at all. Where they do, it is often necessary to resolve data from different sources to one reference sequence so that they can be viewed side-by-side. Different code is needed for each different source and format of reference sequence, and the task is hard to automate due to sequence and transcription variation.
• Unpredictable Data Updates: Often there is no regular update schedule for an LSDB. Moreover there is sometimes no easy way of checking whether an update has occurred. It is therefore difficult to automate updates. At worst the update will be not only to the data, but to the schema, etc. New instances of all of the problems mentioned above may also come into play when reflecting LSDB updates. The process can therefore be time consuming and require more new code to be written.
• The Inability to Query the LSDB: Although making a local copy of all relevant LSDB data can be a viable option (allowing one to create one's own local query engine) in some cases it might be more suitable to query the data on the fly. For instance, in a browser it might be most suitable to get data for the visible genomic region on the fly. Similarly, where local copies of the data are cached it would be nice to query by date in order to retrieve only the data that has not yet been cached. There is no standard way to perform such queries with current LSDBs and it is often impossible.
• Poor Quality Data: LSDB data is often wrong. Errors in numbering will often be immediately obvious in a browser, whereas other errors will not. Web services interfaces would do little to help with this, but standardised interchange formats, LRGs and supporting tools may help improve the numbering problems at least. This would mean that more data could be visualised and potentially that less checking was necessary at the browser end.

These are nearly all problems which are being addressed across Gen2Phen in general. Many of these problems are addressed by the LSDB-in-a-box approach, standardisation and LRGs. However, these do not entirely solve the problem of automatically integrating this data into a browser or automatically keeping it up-to-date.

Required Operations

From the point of view of our relatively simple scenario the operations we would require are along the lines of:

getSummary - IN none - OUT an LSDBSummary
getAllVariants - IN none - OUT a VariantList
getVariants - IN a Query - OUT a VariantList
getVariantByID - IN an ID string - OUT a Variant

For these operations, it is the output types in particular which would depend upon an interchange format. A good deal of work has already been done on the XML interchange format since this text was originally written, so brief notes regarding datatypes are only presented here to clarify the scenario we envision. The real job of building web service interfaces would involve taking the types defined in the interchange format as the starting point.
Query - would allow bounds to be specified on region (reference sequence), time of update and maybe also bounds on region specified in HGVS numbering?? This need not be dependent on the interchange format.

LSDBSummary - would present summary level info. about the LSDB, e.g. name, version, url, link-out-url, creation date, last updated, total entries, number of unique variants, Contact, Gene/s (the latter two would probably be ComplexTypes). A Gene, would for instance include the HGNC symbol and ID, the Entrez Gene ID, the MIM number, the reference sequence used. Personally I think we shouldn't limit the reference sequence to just be a LRG, although they make the job much easier. I think we have to be able to work with legacy data. The LSBSummary would probably be (partly?) dependent on elements of the interchange format.

VariantList = a sequence of Variant objects

Variant = ??? It is this I would really wait for the exchange format on, but it could closely follow the existing PAGE model, the minimal core information suggested by Johan, etc. We don't bring strong requirements in this regard as we are used with working with existing LSDBs. Some reference sequence information and an HGVS name would suffice for us. We are mainly interested in genotype information, and potentially some very "shallow" phenotype information. That is not to say that the format/services should not be designed with a much more generic usage in mind.

Implementation Issues

For performance and dependability reasons we do not want to have our browser as completely dependent on accessing data via external web services. Nor do LSDBs want to be burdened with lots of redundant calls to their services. We would therefore aim to cache LSDB data in a "lazy" fashion. Initially we would get all data from the LSDB and make a local copy. Each time a user browsed to a particular region in a gene we would then make a query for any variants in this region that had been added or updated since the last data of our cached version. If there were any changes we would update our cache. Either way we would update the timestamp on the cache.

One would probably also add further heuristics to avoid making too many calls, such as only checking for updates if the cache is more than X hours old. These differences in strategy don't have any effect on the requirements from the web service interfaces however.

An alternative implementation strategy, which might also be favourable for federation of LSDBs, might follow a publish-subscribe model. In this model clients (including browsers) would subscribe to the LSDB service and the LSDB would then "push" the data to them either whenever there were changes, or according to some schedule. This would have the disadvantage of requiring clients to implement call-back interfaces, which is not always feasible. It could potentially cause problems for LSDBs as well as they might end up with large subscriber lists and not know which were valid (although they could detect whether the call-back interface existed). It also requires some strategy to ensure that clients do not accidentally miss updates. All things considered this seems a more complex option.

We also have to address the question of whether something like DAS would be a better way to address the issue.

Developing Prototype LOVD web services at NGRL

At the National Genetics Reference Laboratory (NGRL), Manchester we are investigating the development of prototype web services to allow machine-to machine access to LOVD data. Here, I hope to provide an overview, and record some of the issues involved as we develop the simplest of these services.

Requirements

The simple service we aim to investigate first of all will provide a means of retrieving all public variants (and any associated public patient data) for a particular gene. This also implies a requirement to be able to get a list of genes supported by a particular LOVD database. For our Browser use case we are also interested in retrieving information about the reference sequence in use.

As this could be the basis of future, more fully featured services, ease of use and ease of installation are also important. Ease of use suggests a REST rather than/as well as a SOAP/WSDL interface. Ease of installation suggests that the service should be developed in PHP, as this will already be present on the target system, since LOVD is developed in PHP.

Implementation Notes

LOVD installation was relatively straightforward. The only problems encountered are well documented on the LOVD website. One of these was to do with strict settings in MySQL.

We investigated PHP Frameworks hoping to find something useful for rapid development of REST and SOAP interfaces and abstraction between the database, PHP classes, XML, etc. We initially looked at WSO2, but quickly found that it was not the quick lightweight solution we needed. It essentially needs rebuilt from source for different platforms, PHP versions, etc. This was not easy and did not meet our requirements. In the end we found the Zend framework to be useful, and have made use of the REST Server in particular.

We found it relatively easy to integrate with the existing PHP code and MySQL tables (centring around the many-to-many relationship between patients and variants) and have started to produce services that reproduce access to the publicly visible LOVD data.

The Web Service

We have so far concentrated on a REST interface to retrieve various information from an LOVD instance in a simple XML format. Because the aim of the exercise is prototyping and exploration we have not supplied an XML schema or stuck to the XML interchange format that is being developed as part of Gen2Phen. A more mature LOVD web service should aim to do this however.

The service is still at an early stage, but you can see the progress (on our own LOVD instance) using the URLs below:

http://ngrl.man.ac.uk/lovd2/ws/rest.php?method=getAllGenes - returns all genes available at that particular LOVD instance
http://ngrl.man.ac.uk/lovd2/ws/rest.php?method=getAllVariants&hgnc_symbol=UBE3A - returns all variants for a particular gene (i.e. as reported in every patient)
http://ngrl.man.ac.uk/lovd2/ws/rest.php?method=getUniqueVariants&hgnc_symbol=UBE3A - as above, but returns unique variants (i.e. one variant element, with 0-n patient sub-elements)
http://ngrl.man.ac.uk/lovd2/ws/rest.php?method=getVariantById&hgnc_symbol=UBE3A&id=UBE3A_00001 - return a single variant based upon the publicly visible database id.

The XML formats returned are more or less what was easiest to produce, and they attempt to reproduce the publicly visible information from LOVD. As you will see we have not dropped any empty optional elements in the results, and are not returning LOVD URLs yet. Below is an example instance of a variant:

	<variant>
		<id>UBE3A_00001</id>
		<exon> 08</exon>
		<dna_change>c.3_16del14</dna_change>
		<rna_change/>
		<protein_change>Frame shift (predicted)</protein_change>
		<restriction_site/>
		<frequency>-</frequency>
		<patient>
			<pathogenicity>
				<reported>Probably pathogenic</reported>
				<concluded>Probably pathogenic</concluded>
				<short>+?/+?</short>
			</pathogenicity>
			<id>003199(MC)</id>
			<disease>Angelman syndrome</disease>
			<reference/>
			<template>DNA</template>
			<technique>SEQ</technique>
			<remarks>Parents not tested - out of frame deletion so pathogenicity assumed.</remarks>
			<times_reported>1</times_reported>
			<variant_created>2008-12-08 16:01:02</variant_created>
			<variant_edited>2009-05-01 16:32:13</variant_edited>
			<patient_created>2008-12-08 16:01:02</patient_created>
			<patient_edited>2009-02-04 12:40:22</patient_edited>
		</patient>
	</variant>

The service is quite lightweight and only requires copying the PHP to your LOVD directory (SimpleXML module is required in PHP, but this is commonly enabled anyway).

Issues

• Many LSDBs do not specify reference sequences, making raw HGVS nomencalture difficult to interpret
• No simple mapping/binding framework from PHP classes to XML was found. The prototype could therefore be hard to maintain as the target XML schema becomes more complex.

Potential Further Work

• Demonstrate visualisation using NGRL browser (this does not require genomic coordinates)

• Rationalise services with other efforts
• Align with XML interchange format and provide feedback on changes to this format

• Get variant by region
• Get variant by feature (5', exon/intron, 3')
• Get variants updated/added since XXX
• Get suggested reference sequence given HGNC gene symbol, list of HGVS variants
• Machine accessible registry service for discovering LOVD web services

Requiring authentication/authorisation:

• Allow viewing of non-public data (e.g. for admin, curators)
• Service to allow submission of data (e.g. from other software)

Exchange format for LSDBs

Hi All

We have been discussing with Ivo about the data format and getting into following proposal:

- Use xml-elemnts and not attributes, due to extendability etc.
- Add separate elements for variation aliases and sequence changes. The latter one is for related, often consequential, sequence and structural changes that are often carried with the main variation entry. Aliases are like db_xrefs but can also have info on reference sequence + other details.

<variant>
 <name>c.34G>C</name>
  <naming_scheme>HGVS</naming_scheme>
  <ref_seq>XY000000</ref_seq>

 ... then detection templates and other usual stuff in a same way

  <aliases>

    <variant>
       <name>c.342G>C</name>
       <naming_scheme>HGVS</naming_scheme>
       <ref_seq>LRG000001</ref_seq>
     </variant>

     <variant>
       <name>MUTXYZ</name>
       <naming_scheme>FINDIS</naming_scheme>
       <ref_seq>NM000001</ref_seq>
     </variant>

  </aliases>

   <seq_change>

      <variant>
         <name>g.232323G>T</name>  <!-- now the change is in genomic DNA -->
         <ref_seq>AC0001</ref_seq>
         <naming_scheme>HGVS</naming_scheme>
      </variant>

      <variant>
         <name>A447RfsX11</name>
        <naming_scheme>FINDIS</naming_scheme>
      </variant>

   </seq_change>

</variant>

============================

Comments:

The main Variation element  is the reference entry people are working with. Basically the sub-variation elements can have same details of data, but this should be optional.

It is not always possible to say should related variation info go into alias or seq_change section. For example variations on different splice variant templates (cDNA templates). But perhaps this does not matter.

Perhaps we should also add  a tag which tells is the seq_change experimentally verified or not.

Implementation specific things like global ids should go into attributes, if those are needed. (?)

<variant id="lsid://findis.org/variant/00001" />


Juha