Aug 8Creating a Local Data Lakehouse using Spark/Minio/Dremio/NessieData is becoming the cornerstone of modern businesses. As businesses scale, so does their data, and this leads to the need for efficient data storage, retrieval, and processing systems. This is where Data Lakehouses come into the picture. What is a Data Lakehouse? A Data Lakehouse combines the best of both data lakes and data…Data Lakehouse6 min readData Lakehouse6 min read
Jul 10Project Nessie: A Look in the DepthsOnce upon a time, in the mystical realm of data lakes, there was a growing problem. The inhabitants of this realm, data scientists, and engineers, were struggling with managing their vast and ever-evolving data assets. It was like trying to organize a library where the books constantly rewrote themselves. …Data Lakehouse6 min readData Lakehouse6 min read
Jun 19Parquet File Compression for Everyone (zstd, brotli, lz4, gzip, snappy)You know how when you’re packing for a trip, you try to stuff as many clothes as you can into your suitcase without breaking the zipper? That’s kind of like data compression in the big data universe. …Data Engineering3 min readData Engineering3 min read
Jun 2Dremio and Modern Data Architecture: Data Lakes, Data Lakehouses and Data MeshToday it can seem like a buzzword onslaught in the data space with terms like Data Mesh, Data Lakehouse, and many more being thrown out with every vendor trying to make their claim that they provide “the answer.” However, in this article, I will discuss the underlying reality that does…Data Mesh5 min readData Mesh5 min read
May 30What is Nessie and Why as a Data Engineer or Architect you should care?We need to establish a few things to understand why the open-source data catalog, Project Nessie, matters so much. Why do Data Lakehouses Matter? The amount of data and use cases for that data are exploding faster than any single tool can develop sufficient support for. However, many tools support different use cases like BI…Data Lake3 min readData Lake3 min read
May 10Resources for Learning more about Catalog level versioning with Project Nessie & Dremio Arctic (Rollbacks, Branching, Tagging and Multi-Table Txns)Data Quality, Governance, Observability, and Disaster Recovery are issues that are still trying to discover best practices in the world of the data lakehouse. A new trend is rising, borrowing from the practices used by software developers to manage these issues with code bases. This trend is called “Data as…Data Lakehouse4 min readData Lakehouse4 min read
Apr 5Overview of the Data Lakehouse, Dremio and Apache IcebergIn this article, I hope clarify the who, what, why, and how of: The Data Lakehouse Dremio Apache Iceberg If you are looking for a more general overview of the data landscape, check out this article I wrote about the data world at a high level. The Data Lakehouse What is the Data Lakehouse? Before the concept of…Dremio6 min readDremio6 min read
Nov 22, 2022Configuring Apache Spark for Apache IcebergApache Iceberg Apache Iceberg is quickly becoming the industry standard for interfacing with data on data lakes. A lot of the time when people first try out Iceberg they do so using Apache Spark. Often to start spark up you may run a command like this: spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:1.0.0\…Apache Iceberg3 min readApache Iceberg3 min read
Aug 9, 20225 Reasons Your Data Lakehouse should Embrace Dremio CloudDremio Cloud makes it easier to enjoy the performance and ease of use often associated with data warehouses. It combines that performance/ease of use with the openness and affordability associated with data lakes, the data lakehouse dream. …Dremio5 min readDremio5 min read
Jul 18, 2022Hands-On Introduction to Apache Iceberg — Data Lakehouse EngineeringAs a Developer Advocate for Dremio I spend a lot of time doing research on technology and best practices around engineering Data Lakehouses and sharing what I learn through content for Subsurface — The Data Lakehouse Community. …Data Lakehouse4 min readData Lakehouse4 min read
