Technical documentation of management systems, data processing and publication tools not specialised on collection data
From GFBio Public Wiki
Revision as of 15:28, 6 November 2019 by David Fichtmueller (Talk | contribs)
Technical documentation of management systems, data processing and publication tools not specialised on collection data at the GFBio Collection Data Centers
One of the goals of GFBio is to strengthen the data centers at the Natural History Collections and Culture Collections in Germany and improve their infrastructure to manage, archive and publish biodiversity research data on the long run. As a consequence – latest after the realisation of the GFBio platform and federal infrastructure – they should be able (a) to mobilise their own data resources for research purposes and (b) to provide their technical infrastructure for managing, archiving and publishing biodiversity research data including multimedia data.
The tools and systems below are set up for management, processing and publishing of data from more than one data domain: collection and observation data, trait data, taxonomic, taxon reference and checklist data, sequence data, sampling data, survey data as well as scientific data packages as a whole. Some are browser-based applications with own online portal, others are usable as desk top stand-alone applications or are part of a client-server network. All are involved in the GFBio dataflow and part of the portfolios of GFBio data centers.
In parallel, the GFBio collection data centers documented their installations of collection management systems, multimedia data management systems, long-term archiving solutions and of GFBio related IT services, tools and databases at the data centers.
| Database system, Homepage | AQUiLA | BacDive | BioCASe Monitor Service (GFBio version) | BioCASe Provider Software | DiversityDescriptions (DD) | DiversityProjects (DP) | DiversityTaxonNames (DTN) | EDIT Taxonomic Editor | Metacat | Morph-D-Base | reBiND | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| General Information | Installations at GFBio collection data centers/ archives | SGN | DSMZ | MfN | BGBM, DSMZ, MfN, SGN, SMNS, SNSB, ZFMK | SMNS, SNSB, ZFMK | SMNS, SNSB, ZFMK | SMNS, SNSB, ZFMK | BGBM | SGN | ZFMK | BGBM |
| Contact persons | A. Schmid and A. Allspach (SGN) | C. Söhngen, A. Podstawka and B. Bunk (Mailcontact of BacDive Team: mailto:contact@bacdive.de) | F. Glöckler and J. Hoffmann (MfN) | J. Holetschek (BGBM) | A. Link (SNSB) | J.C. Monje (SMNS), T. Weibulat and D. Triebel (SNSB) | J.C. Monje (SMNS), D. Triebel (SNSB) and P. Grobe (ZFMK) | A. Müller and K. Luther (BGBM) | A. Niamir and C. Weiland (SGN) | L. Vogt (Uni Bonn) and P. Grobe (ZFMK) | D. Fichtmüller (BGBM) | |
| Developer group, country | Senckenberg IT Services; Biodiversity Informatics SGN | DSMZ IT Services & Bioinformatics Group | Department of Science Data Management at the MfN Berlin, Germany | Biodiversity Informatics BGBM, Germany | SNSB IT Center Germany (together with MfN and UBT as far as the whole DWB platform is concerned) | SNSB IT Center Germany (together with MfN and UBT as far as the whole DWB platform is concerned) | SNSB IT Center Germany (together with MfN and UBT as far as the whole DWB platform is concerned) | BGBM (and Naturalis, NL implementing the Validation Framework) | Knowledge Network for Biocomplexity (KNB), U.S.A. | Biodiversity Informatics ZFMK | Biodiversity Informatics BGBM, Germany | |
| Data domains | collection data, collection management data, observation data | scientific collection data, trait data, physiological, morphological and environmental descriptions | collection data, collection management data, observation data | triple-structured scientific data, e. g. any kind of descriptive or trait data, attribute data, molecular sequence data, sampling data; matrix data import; interactive key functions | scientific project-specific metadata and data package-specific metadata, project settings and identifiers | taxonomic data, taxon reference and checklist data | collection data, taxonomic data (core functionality), molecular data, descriptive data, identification keys | scientific data packages, data sets and data tables, particularly from ecology and environmental science | morphological descriptions, taxonomic data, character matrices; current project: ontology based descriptions | collection data, collection management data, observation data | ||
| Organismic data | all recent and fossil organisms groups; geological data | Bacteria and Archaea (other groups of microorganisms may be in future releases) | all recent and fossil organism groups | all | all | all recent and fossil organism groups | all recent and fossil organism groups (potentially) | all | all | all recent and fossil organism groups | ||
| Software and Database System | Server Operating System | Ubuntu, Apache Tomcat | Ubuntu 12.04 (Apache) | currently on Linux, generally supported by any OS running Apache, SQLite and PHP | MS Windows, Linux, Unix, Mac (any OS that runs Python 2.5, 2.6, 2.7); Also runs inside a Docker container. | MS Windows Server 2008 R2, 2012 R2, 2016; (clients from MS Windows XP to MS Windows 10) | MS Windows Server 2008 R2, 2012 R2, 2016; (clients from MS Windows XP to MS Windows 10) | MS Windows Server 2008 R2, 2012 R2, 2016; (clients from MS Windows XP to MS Windows 10) | Linux, Mac OS, MS Windows | Linux, Mac OS, MS Windows | Linux version 2.6. | currently on Linux, generally supported by Mac OS, MS Windows as well |
| Database system | PostgreSQL | MySQL 5.1 | SQLite | MS SQL Server, MySQL, Access, Sqlite, Oracle, Postgres, 4D, Firebird, Foxpro, Sybase; Excel, CSV | MS SQL-Server 2008 R2 to MS SQL-Server 2016 or MS SQL-Server Express | MS SQL-Server 2008 R2 to MS SQL-Server 2016 or MS SQL-Server Express | MS SQL-Server 2008 R2 to MS SQL-Server 2016 or MS SQL-Server Express | MySQL, PostgreSQL, H2, SQL Server | PostgreSQL (or another SQL92-compliant RDBMS like Oracle) | MySQL, ZOPE ODB | eXist-db | |
| User rights management | Granular user rights assignment | Read rights global, write individual | Granular user rights assignment | On DB level | MS SQL-Server specific; few DWB specific features and roles added | MS SQL-Server specific; few DWB specific features and roles added | MS SQL-Server specific; few DWB specific features and roles added | Roles can be assigned to users according to taxonomic groups and data types (e.g. distribution data only). Project admins have rights to edit all types of data. The underlying software library allows very granular user rights assignment (not yet implemented in the user interface) | Metacat supports internal password file authentication or the use of LDAP as an external authentication mechanism; multiple access right control (specification of rwx rights to single persons, user groups, etc.) in combination with Morpho, a Java-based import wizard for EML which interfaces with the KNB Metacat server | Group based user rights. Entries can be shared read-only or writable within groups. Released entries readable by the public | Group based user rights managment through the eXist-db | |
| Client | Webapplication, Frontend every common webbrowser | Webapplication, Frontend every common webbrowser | Webapplication, Frontend every common webbrowser | Webbrowser | C# desktop application (rich client) (and Web API for the LIASlight project under construction) | C# desktop application (rich client) | C# desktop application (rich client) | desktop application (rich-client), browser based client (for less complex operations, in development) | Browser based client | Browser based client | any modern browser | |
| GIS functionalities | PostGIS extension, Mapserver (WMS/WFS) | - | - | not applicable | - | - | - | Visualization of distribution and point maps | PostGIS extension, Geoserver (WMS/WFS) | None | None | |
| GUIs for data import | generic import tool | - | interactive handling of data | - | Import wizard (DELTA, SDD, CSV), matrix wizard, local import (FASTA, FASTQ) | generic import tool (coming soon); CSV | generic import tool, import wizard; CSV | Currently very simple (but more complex UIs planned) | Webform or Morpho | direct into the database, bulk upload of images possible | Admin interface of eXist-db allows for GUI based upload of files and directory managment | |
| GUIs for data export/ reports | EXCEL, CSV | CSV,PDF, via webservice: XML, JSON | Webservices (JSON) | ABCD and DwC archive export | Export (DELTA, SDD, CSV, FASTA, FASTQ) | various export tools, export wizard (coming soon) | various export tools (e. g. for hierarchy, for accepted names etc.), export wizard; CSV | Currently very simple (but more complex UIs planned) | Export format: EML, Download via Webform or Morpho | CSV, RDF (in development) | None | |
| GUI language | German, English | English | English | English | default: english (multilingual through translation tables) | default: english (multilingual through translation tables) | default: english (multilingual through translation tables) | English and German (Easy to extend for other languages) | English | English | English | |
| Open access | - | - | Yes, internal services designed for GFBio | not applicable | Yes, DD software download | Yes, DP software download | Yes, DTN software download | Yes | Yes | Yes | Yes | |
| Open source | - | - | Yes, MfN GitHub | Yes, BPS SVN code repository | Yes, DWB SVN code repository | Yes, DWB SVN code repository | Yes, DWB SVN code repository | Yes, http://cybertaxonomy.eu/taxeditor/source-repository.html | Yes, (Download source distribution) | Available on request | Yes, Details about reBiND Installation, Prerequisites and SVN code repository | |
| Licenses | - | - | GPL v.3 | Mozilla Public License Version 1.1 | GPL v.2 | GPL v.2 | GPL v.2 | Mozilla Public License Version 1.2 (Project License) | GPL | GPL | Mozilla Public License (MPL) 2.0 | |
| Information model online | BacDive model | not applicable | not applicable | DD data model | DP data model | DTN data model | Common Data Model (CDM) http://cybertaxonomy.eu/cdm/latest/ | - | Coming soon | No | ||
| State of development | ongoing; beta version (search portal) online: https://search.senckenberg.de/aquila-public-search/search, http://www.senckenberg.de/root/index.php?page_id=2868 | ongoing | ongoing (latest release 2.0, https://github.com/MfN-Berlin/BioCASe-Monitor-Service) | ongoing (latest release version 3.7 is http://ww2.biocase.org/svn/bps2/branches/stable) | ongoing (latest stable release see under DWB Software) | ongoing (latest stable release see under DWB Software) | ongoing (latest stable release see under DWB Software) | ongoing (latest stable release Version 5.8, http://cybertaxonomy.eu/download/taxeditor/stable/ released 19.07.2019) | Vers. 2.5.1 (released January, 2016) | Vers. 3.4 | Productive System; development and project itself finished in 11/2014; ongoing maintainance at BGBM | |
| Code language, developer platform | Java, JavaServer Faces, PrimeFaces | PHP, Python, JavaScript | PHP, XSLT, Javascript | Python, JavaScript | C#, .Net Framework, since 2012 | C#, .Net Framework, since 2013 | C#, .Net Framework, since 2005 | Java, Eclipse | Java, JSP | Python | Java, XQuery | |
| User manual | - | Online manual | Online API Doc available on MfN GitHub | BPS Wiki; Beginners Guide | DD Manual as pdf file under DWB User Manuals | DP Manual as pdf file under DWB User Manuals | DTN Manual as pdf file under DWB User Manuals | Integrated into the help system of the software. Online available only older versions. Update is planned. | Online manual (Metacat Administration Guide) | -- | Best Practice Handbook | |
| Training | on demand | on demand | self-explanatory, on demand if necessary; administration manual on MfN GitHub | on demand; User Helpdesk | DWB workshops for users and database administrators; DD included since 2013; User Help Desk | DWB workshops for users and database administrators; DD included since 2013; User Help Desk | DWB workshops for users and database administrators; User Help Desk | on demand; testversion of EDIT Demo-DB with nightly reset, usable with current EDIT version; Handout-EDIT-Platform-Workshop | - | Workshops, individuals on demand | on demand | |
| Notes | - | - | - | - | management of resources (multimedia objects) related to items, descriptors and descriptor states | interface for ingest of GFBio submission information packages (SIPs) (in planning stage) | - | - | - | - | Several data quality modules available to check for common mistakes in ABCD files. | |
| Interfaces to export data in various schemas and standards | xml according ABCD schema | - | - | + | + | - | + | - | + | - | - | + |
| BioCASe Wrapper Version installed, description of dataflow | BioCASe Wrapper V. 3.6.1, underlying db: PostgreSQL view | BioCASe Wrapper V. 3.5, BacDive export under construction | not applicable as the BioCASe Monitor Service connects to several, independent BPS versions | - | - | - | possible on demand | planned | coming soon | custom BioCASe Protocol Endpoint (indepentent of BioCASe Wrapper, full support of the BioCASe Protocol) | ||
| xml according DarwinCore schema | - | under construction | planned for a future version | + | - | - | + | in preparation (currently no priority) | - | not planned | not planned | |
| xml according EML schema | - | - | - | - | - | coming soon | - | in preparation (currently no priority) | yes | not planned | Yes partly, see EML for additional Metadata | |
| xml according other schemas | - | + | not yet, but in future version: any schema supported by the BPS | LIDO; arbitrary xml schemas can be handled | SDD-XML | + | + | CDM-XML. Most data are available through WebServices, most of them as XML or JSON. Documentation available at http://cybertaxonomy.eu/cdmlib/rest-api.html | Transformations from EML to Dublin Core performed by Metacat OAI-PMH produce simple Dublin Core (DC). | DC; RDF in development | theoretically possible with minor additional adjustment/configuration | |
| txt, CSV export | + | + | - | + (DwC-Archive) | + | + | + (export wizard) | DarwinCore-Archive (csv incl. metadata), Excel (for taxonomic core data) | - | csv | - | |
| Notes | - | - | - | Can be run inside a Docker container. | several document generators (html, MediaWiki format) and pipelines for dynamic web publication; local import and export of molecular sequence data in FASTA and FASTQ format, DELTA export for use by NaviKey installations; see also DWB network and installations at SNSB; DWB network solutions; DD is a suitable tool for conceptual schema development as done with the MOD-CO schema; see also DiversityDescriptions Quiz Version | - | DTN managed open data (taxon reference lists and checklists) are in the DWB cloud at SNSB IT Center and available through DTN REST Web Service; documentation under http://services.snsb.info/DTNtaxonlists/rest/v0.1/static/api-doc.html | Remote editor included since Version 3.11; browser based editor (to improve usability) currently available for occurance data; development ongoing and close related to the overall development of the TaxonomicEditor. | The establishment of a Metacat repository does not mean that the database system automatically has a connection to DataOne. SGN will manage and archive the GFBio compliant EML-structured xml-files completely independently (though membership to DataOne might be considered.) | - | - |
Status: July 2019
Back to Technical Documentations
