- This line was added.
- This line was removed.
- Formatting was changed.
In this section you can find only the main top-level features scheduled for next releases. For a more detailed list you can go to GitHub Issues at https://github.com/opencb/opencga/issues
OpenCGA 2.x Planned Releases
2.0.0 (Oct 2019)
Still work in progress, this is the tentative list of features.
- Improve Audit system, make all actions audited and queryable
- Implement a new Notification system, this will allow other applications to know what's going on in OpenCGA
- Implement a centralised log solution, the initial plan is to use Kibana for this feature
- Implement a new Cache system, the plan is to use Redis to cache some queries
- Improve Docker images solution, now users can easily run from Docker Hub any stable version
- Upgrade several dependencies: Java 11, MongoDB 4.x, Solr 8.x, JUnit 5.x
- Clean ups and remove deprecated code and APIs
- Add ACID Transactions to Catalog
- Improve REST web services return types and add error codes to the response, this will improve debugging
- Initial support of FIHR, in this release we will start supporting some data models and making OpenCGA compatible with FIHR. In the next release we aim to provide a full implementation of FIHR.
- Prepare OpenCGA Catalog for supporting Federation
- Improve performance of most queries
- Develop an Analysis framework to allow users to customise and implement their own analaysis
- Add Wrapped analysis such as Plink, RVtests, ... as by default analysis
- Support CRAM files
Variant Storage Engine
- Implement variant imprecise queries and some specific structural variant queries
- Remove any blocking operations, any operation should be able to run at any time
- Add native variant analysis such as GWAS, Sample Stats, sex inference, ... using MapReduce or other computing frameworks such as Apache Spark
- Improve HBase sample index performance by improving covered queries
- Implement HBase-based aggregations
- Improve testing of variant query and analysis
- Implement Cancer Tiering interpretation analysis algorithm
- Implement a Clinical pathogenic database for all reported variants
- Network-based clinical interpretation algorithm
- Implement Secondary Findings analysis
- Full support for Microsoft Azure and HDInsight 4.0, this also includes Azure AD, Azure Blob and Azure Batch. We would like to thank very much Microsoft Azure for their amazing support and help here.
- Add Kubernetes to deploy and orchestration
Note: some of these features might be released in the Enterprise version coming soon
OpenCGA 1.x Planned Releases
1.4.0 (February 2019)
- Implement the new HTSGET 1.0 protocol
- IVA 0.9.0 will implement a full study and clinical analysis among many other features
- Add many more negative and variant functional tests
- Documentation improvements with new diagrams and tutorials
- Complete and test all delete operations and implement delete by queries to make easier to delete batches of resources, with this the REST API can be considered complete
- Implement a new admin REST API, this will allow OpenCGA administrator to execute administrative tasks remotely
- New PermissionRule feature, you can define rules for assigning permissions automatically when new data is created, e.g. set VIEW permission to USER to all samples where HOSPITAL = 'X'
- New implementation of how clinical data (annotation sets) are store in the database, this new physical schema significantly improves querying annotations (even with nested objects or arrays), group by aggregations, include/exclude filtering and allow to flatten the annotations
- Complete ClinicalAnalysis, ClinicalInterpretation and ClinicalReport data models and functionality
- Support the last pending structural variant: Translocation. With this all structural variants are properly represented and stored
- Improve variant stats and add simple variant analysis such as association or Hardy-Weinberg test, this will be stored and indexed in the new VariantScore object
- Add INDEL left-alignment normalisation to VariantNormaliser
- Variant Benchmark suite to study scalability and performance
1.3.0 (November 2017)
- CLI autocompletion implemented
- New single CLI for execute migrations automatically
- New and fully functional R client library for REST web services, with this the four client libraries are completed
- New IVA 0.9.0 is developed coordinately to exploit all the new features, they will be released together
- Many more functional tests added to test all new functionality described below
- Review and improve Swagger documentation and descriptions
- Documentation improvements with new diagrams and tutorials
- New Family data model finished, now it is production ready, this completes and integrates three related data models: Sample, Individual and Family
- New Versioning feature implemented for Sample, Individual and Family. Now you can track any change in those data models, users can query o review any version of those documents
- New Export functionality implemented, this allows to export a Project as it was at any specific release, this can then imported in a new OpenCGA server
- New Study administrative group called admins, all users in this group will be granted some special permissions at Study level such as create groups or share data, this will make Study administration much easier
- New Confidential permission for Variable Sets, now you can make some clinical data private for some users
- New ClinicalAnalysis data model added, this allows to define and stored different clinical interpretation analysis, this is still experimental and it should not be used in production
- Improvements in Group By queries, now you can pass a count parameter and aggregations only use data you can view, this can be useful for summarising data. Also, this has been added to Individual and Family
- Ensure that all query GET REST web services accept comma-separated list of IDs, at the moment only few of them accept ID lists, this will reduce the number of REST calls needed improving the performance
- New REST web service to execute remote scripts for Catalog, for instance "move samples from Study"
- Performance improvements when checking permissions (ACL) in create and update methods, now on average 50% less database queries are needed
- Improve support for Structural Variants, in this release we will fully support Insertion, Deletion and Copy Number variants
- New VariantMetadata implemented, this is exported together with the variant data to be further analysed with other OpenCB projects using Spark
- New VariantScore object added to Variant data model, this will allow to store variant scores from cohort-related analysis such as association or Hardy-Weinberg tests in the next release
- Implement some HBase physical schema improvements and a better integration with Solr
- Support Amazon EMR Hadoop cluster
- Performance improvements when querying variants from samples, this will have a big impact in clinical interpretation analysis
- Major improvements in BAM query engine. New server-side filters added, this is a more efficient implementation since the data sent through the network is reduced. The available filters now are: region, minMapQ, maxNumberMismatches, maxNumberHits, properlyPaired, maxInsertSize, unmmapped and duplicated.
- New coverage calculator using BigWig. Now coverage is calculated and stored in BigWig format, the windowSize is configurable. Also, coverage can now be queried for a region and optionally a windowSize, the server will aggregate and compute the average in windowSizes.
- New REST and gRPC APIs implementing the new query filters and coverage functionality. When using REST a JSON string is returned using GA4GH data model. When gRPC is used a binary stream is obtained. Note that in both protocols the filters are applied in the server.
The following features have been accepted but no release version has been assigned:
- Add test for the CLI
- Support Slurm
- Add Reactive Programming (RxJava) and Events, this will allow to be easily integrated into other custom Java-based applications
- New Gene Expression database, this will include a Gene Annotation based on CellBase
You can find detailed information for some of them at https://github.com/opencb/opencga/milestone/10
Table of Contents:
|Table of Contents|