Hello!

The fall-winter season has been an exciting time at TileDB. We received strategic investments from Lockheed Martin Ventures and NTT Docomo Ventures and continued to grow our team. As we close out 2021, here’s an update on the progress we’ve made on TileDB Cloud, TileDB Embedded, and more.

We’ve been working to make it easier to extend your work within TileDB Cloud. Whether that’s exposing notebook-powered visualizations to the web, to managing & versioning ML models and more, these new features are already available in the platform.

Dashboards

Notebooks on TileDB Cloud provide a convenient space to experiment with data and scale computations. Now, the same notebooks that power your research workflows can expose data visualizations to the web as interactive dashboards.

Dashboards are a dedicated asset type in TileDB Cloud that can be shared privately or publicly. Public dashboards are discoverable via the Explore tab.

New asset types: ML models & files

You can now centrally train and manage ML models on TIleDB Cloud, including plugins for PyTorch and TensorFlow. Sharing works the same way you already share arrays, notebooks and other assets.

We’re also pleased to introduce file management to further your research and collaboration projects. TIleDB Cloud represents all assets as arrays under the hood, and this now applies to arbitrary files too, allowing files to inherit the versioning, time traveling and sharing features built into TileDB arrays.

Asset versioning

In our last newsletter, we described upcoming time-traveling functionality for code changes to TileDB Cloud notebooks and UDFs. In addition to arrays, we are happy to announce that versioning and time traveling are now live for the following asset types in the TileDB Cloud UI: notebooks, UDFs. ML models and files.

Here's what's coming next.

JavaScript API

Soon, you’ll be able to query arrays registered to TileDB Cloud directly from the web browser. A demo app is available on GitHub, showing how to create a simple 2D array and edit its attributes. Look for more updates on the TileDB Cloud JavaScript API in 2022!

GPUs

We are working to make GPU-backed code execution available for both TileDB Cloud notebooks and UDFs. GPUs will be enabled as an additional option on the “Launch Notebook” tab. For UDFs on GPUs, we are currently planning this implementation. If you have requirements for high-performance UDFs, please reach out with feedback.

Groups

Groups are an original part of the design of TileDB Embedded, allowing users to organize arrays into folder hierarchies. Now, groups will soon become first-class citizens on TileDB Cloud, with full UI support for both data discovery and sharing. Watch the TileDB Cloud console for more updates on groups, coming soon.

Start with TileDB Cloud today

Highlights from versions 2.4, 2.5, 2.6.

Schema evolution

TileDB 2.4 introduced new schema evolution features for dense and sparse arrays. This first release allows for versioning of schemas and for the ability to add and drop columns. Previously dropped columns can be restored until arrays have undergone consolidation, which is the process that flattens fragment metadata.

Performance improvements

In the upcoming release of TileDB Embedded 2.6, you will notice that we refactored many of the core data processing operations, which will vastly improve parallelization of tile bitmap computations, Zstd filter pipelines, and more. The 2.5 release introduced an update to the core TileDB library, which optimizes queries where sorting and deduplication are not required. In these cases, we have seen 10x faster read performance.

New CLI

Alongside 2.5, we released new versions of the TileDB-CLI to improve working with array fragments.

TileDB was recently interviewed on the Cloud Database Report Podcast. Hosted and produced by veteran editor and industry analyst John Foley, TileDB Founder & CEO Stavros Papadopoulos discussed how a universal database based on multi-dimensional arrays supports a variety of data types for genomics, geospatial and many other analytics use cases. Give it a listen and subscribe to the podcast for more great interviews.

We’ve been working to create more educational video content. Check out events.tiledb.com for our 2021 webinars, where we covered TileDB Embedded, TileDB Cloud, Population Genomics and LiDAR. If you missed us at a recent conference, no worries! We have collected talks from FOSS4G, useR!, and many other events on Youtube.

And there’s more to come! Please let us know what you'd like us to cover next by emailing [email protected].

See you in 2022!

Extending our warmest wishes for the holiday season and a wonderful new year. All the best!

Thank you,

— The TileDB Team