Page tree
Skip to end of metadata
Go to start of metadata

OpenCGA is still under a very active development, we are heading to the first stable release in 2017 Q2, you can find more detailed information in GitHub Releases at https://github.com/opencb/opencga/releases.

1.4.0-rc1 (August 2018)

Catalog

  • [IMPROVEMENT] Recognise bigwig files automatically. (#283)
  • [IMPROVEMENT] Major improvements in annotationSets (#635, #772, #849):
    • Queries can be performed by any of the fields. (#772)
    • All the fields are indexed in the database, supporting really fast queries. (#772)
    • Projections can be added so only specific fields of an annotationSet are included. (#635, #772)
    • Annotation set web services have been changed. (#849)
    • See AnnotationSets 1.4.0 section from the OpenCGA documentation for the whole description of features.
  • [FEATURE] Implement new "Permission rules" feature (#745)
  • [FEATURE] Add new /admin web services to be able to perform administrative operations through REST (#759)
  • [IMPROVEMENT] Improve /user web services. Remove duplicated configs web services (#661)
  • [FEATURE] Implement delete operations for most of the entries (#792)
  • [IMPROVEMENT] Store creationDate field as an actual date object to improve queries (#752)
  • [CHANGE] Important id changes (#819):
    • Data model ids have been redefined to contain any user-defined id for every entry. Migrations produced in every entity are described in the ticket.
    • Old numeric ids are now lost and only used for internal purposes. Users cannot query by those fields anymore.
    • Update REST web services to support id changes (#834)
  • [IMPROVEMENT] Stop creating empty directories when linking Files (#865)
  • [FEATURE] Add new {entry}Action field to some update web services to support adding, setting or removing single entries to arrays of them. (Example: samplesAction to add, set or remove samples from an individual, file or cohort) (#850)
  • [IMPROVEMENT] Deprecate VariableSet web services and move them to Study web services. Improve and clean old Study web services (#846)
  • [IMPROVEMENT] General data model improvements described in #823:
    • Add new inmutable uuid field to all entries.
  • [FEATURE] Create catalog solr sync mechanism and enable facet queries #875

Variant Storage

  • [FEATURE] Aggregate operation for all samples from a study in the Variant (#757)

    • [IMPROVEMENT] Move HBase write step to a separated MR for fill-missing operation (#815)

    • [IMPROVEMENT] Mark already processed variants in fill missing operation (#803)

    • [IMPROVEMENT] Do not iterate over all VcfRecords when filling gaps or missing variants (#794)

    • [IMPROVEMENT] Fill missing prepare step: copy variants to fill into archive table (#793)

    • [IMPROVEMENT] Split VcfSlice from archive table in REF and NON_REF columns (#778)

    • [IMPROVEMENT] Split archive table in file batches (#777)

    • [CHANGE] Remove merge=advanced from storage-hadoop (#796)

  • [FEATURE] Index Sample Genotypes in HBase (#838)

    • [FEATURE] Genotype index intersect in HBase (#862)

    • [FEATURE] Integrate Variants export using MapReduce (#867)

    • [FEATURE] Support enriched genotypes when querying to SampleIndex (#870)

    • [IMPROVEMENT] Use variants SampleIndex when reading from MapReduce (#868)

  • [FEATURE] New Variant query filter "clinicalSignificance" (#872)

  • [FEATURE] Add new filters for INFO and FORMAT fields other than GT (#863)
  • [FEATURE] Add filter by QUAL in Storage Variants (#809)

  • [FEATURE] Accept enriched genotypes at GENOTYPE filter (#750)

  • [FEATURE] Variant filtering in a samples group (#578)

  • [CHANGE] Automatic INCLUDE_STUDY if possible (#878)

  • [CHANGE] Automatic INCLUDE_SAMPLE if GENOTYPE VariantQueryParam is present (#814)

  • [IMPROVEMENT] Match Consequence type with the specified Gene filter (if any) for Storage Hadoop (#874)

  • [CHANGE] Untie internal Catalog and Storage IDs (#859)

    • [FIX] Synchronize Catalog ID to UID changes in Storage Managers catalog (#861)

  • [FEATURE] Create a ProjectMetadata object for Variant Storage (#832)

  • [IMPROVEMENT] Move storage-hadoop metadata to a separated table (#781)

  • [FEATURE] Support and load BREAKENDS in Variant Storage (#760)

    • [FEATURE] Support of Symbolic and Structural variants in Hadoop Variant Storage (#857)

  • [IMPROVEMENT] Add direct loader for the first file in a chromosome to load directly to the variants collection (#354)

  • [FEATURE] Move removed variants into a trash bin (#831)

  • [CHANGE] By default, load all FORMAT fields in Variant Storage MongoDB (#824)

  • [CHANGE] Change storage-hadoop table naming policy (#782)

  • [IMPROVEMENT] Review and improve Variant Annotation in OpenCGA Storage (#805)

  • [IMPROVEMENT] Create one Stage Collection per study in Variants Storage MongoDB (#801)

  • [IMPROVEMENT] Filter out overlapping files in variants in storage-hadoop (#779)

  • [IMPROVEMENT] Improve remove file in variant-storage-hadoop (#776)

  • [IMPROVEMENT] Load annotation using BufferedMutator (#775)

  • [FIX] Do not store phoenix primary key columns in HBase as separated columns (#802)

  • [FIX] Adapt remove file to new archive schema (#800)

  • [FIX] Do not map namespace for VIEW tables under phoenix 4.12.0 (#799)

  • [FIX] Avoid PhoenixIOException timeout when dropping columns from phoenix (#795)


1.3.8 (August 2018)

Catalog

  • [FIX] Fix permission issue affecting users and groups with the "_" symbol (#881)

1.3.7 (July 2018)

Catalog

  • [ENHANCEMENT] Add new tags field to the File data model (#855)
  • [CHANGE] Configuration change. Add hooks to configuration file (#856)

1.3.6 (May 2018)

Catalog

  • [FIX] Improve performance of sample queries filtering by individual (#843)

1.3.5 (May 2018)

Catalog

  • [FIX] Fix issue when assigning permissions given the id(s) of different entities  (#836)

1.3.4 (April 2018)

Catalog

  • [ENHANCEMENT] Performance improvement when assigning permissions  (#829)

1.3.3 (March 2018)

Catalog

  • [ENHANCEMENT] Remove old deprecated fields from Family data model (#810)
  • [ENHANCEMENT] Allow looking for Individuals and Families by a new sample field (#811)
  • [FIX] Fix command line to support non-expiring tokens
  • [FIX] Support SKIP_COUNT parameter
  • [CHANGE] Remove family completeness check

1.3.2 (February 2018)

Catalog

  • [FIX] Add missing individual-sample indexes (#790)
  • [FIX] Fix smart name resolution (#791)

1.3.1 (February 2018)

Catalog

  • [FIX] Propagation of permissions sample-individual not working (#780)
  • [ENHANCEMENT] Ask for admin password automatically when using admin command line (#785)
  • [FIX] Filtering by file size not working (#786)

1.3.0 (January 2018)

General

  • [FEATURE] Admin migration command line (#690)
  • [FEATURE] Implement AutoComplete for CLI (#714)

Catalog

  • [REMOVE] Remove ACL from data models. (#666)
  • [ENHANCEMENT] Clean old code and refactoring. (#667, #668, #669, #670)
  • [ENHANCEMENT] Remove some hidden and deprecated methods from webservices and command line (#672)
  • [CHANGE] Changes to family data model (#677)
  • [FEATURE] Add new analysis tool webservices (#679)
  • [FEATURE] Add version support for SampleIndividual and Family (#684)
  • [ENHANCEMENT] Clean and remove unnecessary dependencies for the client module (#687)
  • [ENHANCEMENT] Change some fields from Clinical Analysis (BETA) data model (#688, #702)
  • [ENHANCEMENT] Remove unnecessary Relatives data model (#693)
  • [ENHANCEMENT] Improve some individual webservices to better support the Individual-Sample relation (#701)
  • [CHANGE] Change individual parameter in the sample/create webservice (#703)
  • [CHANGE] Internal modification regarding the way the Sample-Individual relation was stored (#706)
  • [ENHANCEMENT] Add new admins group in studies (#711)
  • [ENHANCEMENT] Add new stats field to Sample data model (#717)
  • [CHANGE] Rename ontologyTerms field in Sample, Individual and diseases field in Family for phenotypes (#718)
  • [FEATURE] Add new option to export and import data from/to catalog (#720)
  • [ENHANCEMENT] Improve groupby webservices (#721)
  • [ENHANCEMENT] Support a list of ids in all GET webservices (#727)
  • [CHANGE] Internal modification: Change ACL delimiter used (#740)

Variant Storage

  • [FEATURE] Make use of the new VariantMetadata model from Biodata (#673)
  • [FEATURE] Major support of Symbolic variants in Variants Storage (#695)
  • [FEATURE] Create profiles to select hadoop flavour (s#707)
  • [FEATURE] Store info fields on storage-hadoop improvement storage (#704)
  • [FEATURE] Allow loading multiple variant files from the same sample with non overlapping variants (#696)
  • [FEATURE] New optional pipeline step "fill-gaps" (#713)
  • [CHANGE] Rename some Variant REST query parameters (#751)
  • [CHANGE] GO and EXPRESSION filter must be combined as an AND with other region (#694)
  • [CHANGE] Update CellBase to v4.5.3 improvement (#770)
  • [ENHANCEMENT] Speed up GENOTYPE filter with FILES filter, when possible (#675)
  • [ENHANCEMENT] Add field "source" to VariantQueryResult (#758)
  • [ENHANCEMENT] Indicate if "numTotalResults" is an approximated count in VariantQueryResult (#749)
  • [FIX] Inconsistent configuration param to select variant annotator (#747)
  • [FIX] Duplicate Key Warn/Error in Stage Collection (#766)
  • [FIX] IllegalArgumentException when CellBaseRestVariantAnnotator skips a variant (#746)
  • [FIX] Concurrent table modification error when loading in Hadoop with merge=basic (#709)

1.2.0 (September 2017)

Catalog

  • [FIX] Fix job search by input and output files. (#533)
  • [ENHANCEMENT] Hide deprecated webservices. (#599)
  • [ENHANCEMENT] Set ontologyTerms array during Sample/Individual /create and /update webservices. (#613)
  • [FEATURE] Add the concept of release in OpenCGA. (#616)
  • [FIX] Fix behaviour where individual ids are not recognized when creating new family. (#617)
  • [FEATURE] Implement JWT based session management (#618)
  • [ENHANCEMENT] Change List<Long> for List<Object> in corresponding data models (#621)
  • [ENHANCEMENT] Make ACL permissions part of the query (#628)
  • [FIX] Fix count parameter not working in search webservices and command line (#629)
  • [CHANGE] Change annotationsets REST webservices. (#631)
  • [FEATURE] Support multigroups (#633)
  • [ENHANCEMENT] Return HTTP 403 error code when user tries to access not granted data (#636)
  • [FEATURE] Add a new members group for every study (#642)
  • [FEATURE] Automatically sync OpenCGA groups with external LDAP groups during login (#647)
  • [FEATURE] Add new projects/search webservice (#651)
  • [FEATURE] Add new confidential parameter to VariableSet data model. It allows defining confidential variable sets and corresponding annotation sets, so a new special permission will be needed to access them. (#653)
  • [FEATURE] Propagate permissions from samples to individuals. (#657)
  • [ENHANCEMENT] Return HTTP 401 error code when user is not successfully logged in or the token is invalid. (#658)

Variant Storage

  • [FEATURE] Improve Solr integration with VariantStorage when querying (#638)
  • [ENHANCEMENT] Improve Solr variant iterator by using Solr cursors (#640)
  • [ENHANCEMENT] Variant Solr Search manager improvements (#639)
  • [ENHANCEMENT] Configure VariantMerger and VariantNormalizer with VCFHeader (#630)
  • [ENHANCEMENT] Store DisplayConsequenceType from VariantAnnotation at MongoDB (#659)
  • [ENHANCEMENT] Return VariantTraitAssociation as TraitAssociation (EvidenceEntry) (#692)
  • [FEATURE] Remove files from variants storage (#192)
  • [FEATURE] Command line and rest endpoints for remove operations on variant storage (#623)
  • [FEATURE] Simple merge mode for loading variants in opencga-storage-hadoop (#609)
  • [FEATURE] VariantQueryParam INCLUDE_FORMATS. Select format fields to return (#608)
  • [FEATURE] Store other genotype fields on storage-hadoop (#602)
  • [FEATURE] Export variant statistics (#309)
  • [FIX] Sample filter not working when the sample is in multiple files (#641)
  • [FIX] Possible error loading overlaping multiallelic variants (#626)
  • [FIX] Do not delete files (or related entries) from catalog if still loaded in variants storage (#625)
  • [FIX] Avoid OutOfMemoryError updating storage metadata from catalog (#645)
  • [FIX] Error indexing vcf files containing "variants" in the file name(#691)

Relevant changes 

1.1.0 (June 2017)

Catalog

  • [ENHANCEMENT] Support integers and floats type for variables. (#545)
  • [BUG] Fix link race condition. (#551)
  • [FEATURE] Add new parameter propagate to Individual web service when setting permissions to propagate permissions to the related samples. (#558)
  • [FEATURE] Add support to give permissions using queries in sample, individual and file web services. (#560)
  • [CHANGE] Change Acl REST web services. (#561)
  • [FIX] Not return fields that are of no interest (using include/exclude). (#569)
  • [FEATURE] Add new admin command line to synchronise and add users from LDAP groups. (#573)
  • [FEATURE] Added new Family data model. (#582)
  • [FEATURE] Add list<Sample> to individual/create web service. (#583)
  • [FEATURE] Add new /meta/status, /meta/ping and /meta/about web services. (#572)
  • [ENHANCEMENT] Support creating an Individual when calling to the Sample create web service. (#586)
  • [DEPRECATE] Deprecate species field of Individual data model. (#588)
  • [ENHANCEMENT] Deprecate usage of variableSetId. Add field variableSet to corresponding web services that accepts an id or a name. (#589)
  • [ENHANCEMENT] Support passing an array of annotationSets when creating an Annotable entry (Sample, Cohort, Individual and Family). (#590)
  • [ENHANCEMENT] Add new type field to Sample model. (#591)
  • [FEATURE] Add father and mother information in attributes field of Individual. (#592)
  • [ENHANCEMENT] Add support to change public/private User registration  (#594)
  • [DEPRECATE] Deprecate all xx/create and xx/update GET webservices. (#598)

Variant Storage

  • [FEATURE] Make optional to merge non same overlapping variants in MongoDB (#574)
  • [FEATURE] Implement a benchmark framework for OpenCGA Storage (#248)
  • [ENHANCEMENT] Filter VcfRecord before converting into Variant object when possible (#577)
  • [BUGFIX] Not loading new overlapping variants in HBase (#581)
  • [ENHANCEMENT] Increment variant size threshold for CellBase annotation (#596)
  • [BUGFIX] Fix ArrayIndexOutOfBounds when loading variants data (#597)
  • [FEATURE] Add VCF export to gRPC command line (#606)

Relevant changes - migration

  • Add new permissions to admin user in the general configuration file. To add: VIEW_STUDY, UPDATE_STUDY and SHARE_STUDY
  • Added sampleIds parameter in each individual entry. 
  • Catalog changes require this migration script: opencga-1.1.0-migration.js
  • Changes of #574 require to execute a migration script on all Variants databases in MongoDB: opencga_574_add_studies_field_to_stage.js


1.0.2

General

  • [BUGFIX] Fix VCF output format (#584)

Catalog

  • [ENHANCEMENT] New dateOfBirth field added to Individual (#580) and somatic field to Sample (#576)
  • [ENHANCEMENT] Performance improvement when annotating new variants (#575)


1.0.0 (February 2017)

Catalog

  • [FIX] Authenticated users can now see public data (#501)
  • Permissions assigned to individuals are directly propagated to samples (#509)
  • [CHANGED] CREATE and UPDATE permissions have been merged into WRITE (#506)
  • [CHANGED] Permissions given to folders are now propagated in the database (#505), this increase significantly the performance of the ACL resolution 
  • [CHANGED] Changes in session data model (#479)

Storage

  • Complete the implementation of exclude and include of fields for Variant queries (#515)

Clients

  • [Python] new Python client implemented following the same architecture than Java and Javascript clients (https://github.com/opencb/opencga/pull/516). This improves the quality significantly and add many new features, thanks Antonio Rueda!
  • Java and Javascript client libs use always POST when available

Server

  • [REST] Add new files/create using POST to create new folders and files with some content (#514)
  • [REST] Rename parameters from acl/update (2617993)
  • [REST] Hide from Swagger all create and update ACL web services using GET (cbea817), these should always use POST, this affects to studies, samples, files, ... (these will be removed in version 1.1) 
  • [REST] Complete the implementation of exclude and include of fields for Variant queries
  • [REST] Remove from users the GET methods to create and update filters (5125a22)
  • [REST] Add a warning to Swagger to all create and update methods using GET (ee6d66). It is encourage that all create and update actions use always POST
  • [GRPC] Complete the implementation of exclude and include of fields for Variant queries

1.0.0-rc3 (January 2017)

General

  • Major changes in maven properties and configuration files. (#480)
  • catalog-configuration.yml and configuration.yml have been merged. (#476)

Catalog

  • Added support to fetch shared projects and studies.
  • Added organism information to the project data model. (#455)
  • Renamed diskUsage and diskQuota for size and quota respectively in all data models were present.
  • Closed sessions are now removed from the array of sessions (but they can still be found in the audit collection). (#475)

Storage

  • Improve error handling for storage operations (#447#463#465)
  • Add param --resume to opencga-analysis.sh and opencga-storage.sh command line (#465)
  • Import and export variants datasets (#459#460)
  • Enable gRPC as top level feature (#492)
  • New top layer StorageManager connecting catalog with storage (#486)
  • Shade proto and guava dependencies for Hadoop (#440)

Server

  • Create and update webservices have been all implemented via POST.

Known issues

  • OpenCGA storage hadoop is not available in this version. To compile use this line:

    mvn clean install -DskipTests -pl '!:opencga-storage-hadoop-core'

1.0.0-rc2 (November 2016)


1.0.0-rc1 (September 2016)

This release constitutes the first release candidate (RC1). This is the biggest release ever with more than 1,400 commits, special mention to j-coll and pfurio for their contribution in Catalog, Storage and Server components.

General

  • New command line interfaces (CLI) for users (opencga.sh), admins (opencga-admin.sh) and analysis (opencga-analysis.sh)

Catalog

  • New catalog configuration file using YAML.
  • New authorisation method. A big list of permission have been defined.
  • New smart id resolver. Numerical ids are no longer mandatory and ids are now resolved internally given the alias.
  • New and improved java and python command lines.
  • New javascript, java and R client implementations.
  • Support for annotations in cohorts.

Storage

  • Many performance improvements and fixes in MongoBD storage engine, the most notable include a new load strategy to improve file merging scalability
  • New storage engine based on Apache HBase, this is completely functional but is tagged as beta until more tests are done.
  • New custom variant annotation implemented

Server

  • An experimental gRPC server implemented


v0.7.0

Third Beta

v0.6.0

Second Beta

v0.5.0

First Beta


  • [IMPROVEMENT] Add direct loader for the first file in a chromosome to load directly to the variants collection (#354)


Table of Contents:


  • No labels