Czech the Data

I'm Mike Czech, a software engineer and data scientist living in Hamburg, Germany. I work on autonomous driving safety at MOIA and write about data, machine learning, software engineering, travel, and other things I’m learning.

Jun 11, 2023

Reducing Memory Requirements with Sparse Data Structures

Using sparse data structures can vastly reduce memory requirements. Sparse data refers to data where most entries are zero, while dense data contains a higher proportion of non-zero values.

[... 283 words]

Jan 12, 2023

BigQuery SQL Query Testing with DuckDB and SQLGlot

SQL plays a pivotal role in data management, yet its testing phase often encounters challenges, particularly due to dependencies on large-scale systems like BigQuery or Hadoop. This dependency not only complicates the process but also slows it down.

[... 365 words]

Sep 20, 2022

Reconsidering Data Types for Efficiency

[... 242 words]

data-engineering

Mar 20, 2022

Columnar vs. Row-Based Storage: Key Differences and Use Cases

If you encounter slow SQL queries or data processing in general, it might indicate a problem with your data and how it’s stored. Even if the delay is only a few minutes, the iterative and exploratory nature of working with data can cripple productivity (and sanity).

[... 347 words]

Jan 22, 2022

Enhancing Code Performance with Vectorization

Python has the reputation of being slow and often rightly so. One of the reasons why this language is still so popular for a compute-heavy field like machine learning is that most libraries are actually written in a faster language like C, C++, or Rust.

[... 351 words]

Nov 04, 2020

Table Partitioning and Clustering Strategies

A classic performance optimization when dealing with larger amounts of data relates to typical usage patterns: Like requiring only a subset of columns, we often want to only use a subset of rows. For example, we might only consider a given time period like all the rows from the last three months.

[... 283 words]

Sep 13, 2020

Implementing a Python Package Repository on GCP

Update (March 14, 2021): Google has recently announced that their Artifact Registry now also supports PyPI repositories in its alpha stage (refer to Artifact Registry documentation for more information). While I haven’t had the opportunity to test this feature personally, it appears to be a promising development.

[... 832 words]

Apr 15, 2019

Optimizing Resource Utilization in Batch Jobs on GKE

In our team, we frequently execute batch jobs that demand specialized hardware, particularly model training tasks that typically require multiple GPUs for timely completion. To facilitate these jobs, we employ the Google Kubernetes Engine (GKE), a managed service on the Google Cloud Platform (GCP).

[... 567 words]

Jul 08, 2014

Concolic Testing With KLEE

Software testing is a complex and challenging task. While complete automation of this process remains a distant goal, there are promising techniques emerging in the field.

[... 452 words]

testing