article Apr 11, 2025A First Glance at Python UDTFs in SnowflakeRecently, I’ve been working more with dbt / Snowflake and needed to utilize a multi-output regression model from SQL. The model was implemented in Python using scikit-learn and LightGBM.[... 663 words]pythonsqldata-sciencedata-engineering
article Mar 02, 2025A Lightweight Vector DB with DuckDB and Cloud RunWhile working on a project, I recently needed a lightweight, serverless vector database solution. The vectors needed to be stored alongside additional metadata, and the database itself would only be updated daily—i.e., read-only for most users.[... 518 words]data-engineeringcloudgcppythonduckdbvector-db
article Oct 11, 2024Do you really need a distributed query engine?At first glance, using distributed computing sounds like a great idea. What’s better than using a fast computer?[... 362 words]data-engineeringperformance-tuning
article Jun 11, 2023Reducing Memory Requirements with Sparse Data StructuresUsing sparse data structures can vastly reduce memory requirements. Sparse data refers to data where most entries are zero, while dense data contains a higher proportion of non-zero values.[... 283 words]data-engineeringdata-scienceperformance-tuningmachine-learning
article Mar 20, 2022Columnar vs. Row-Based Storage: Key Differences and Use CasesIf you encounter slow SQL queries or data processing in general, it might indicate a problem with your data and how it’s stored. Even if the delay is only a few minutes, the iterative and exploratory nature of working with data can cripple productivity (and sanity).[... 347 words]data-engineeringperformance-tuning
article Nov 04, 2020Table Partitioning and Clustering StrategiesA classic performance optimization when dealing with larger amounts of data relates to typical usage patterns: Like requiring only a subset of columns, we often want to only use a subset of rows. For example, we might only consider a given time period like all the rows from the last three months.[... 283 words]data-engineeringperformance-tuning