I thought I was going to build a data platform. I didn’t know I was also going to build my own way of thinking about platforms.

Six months ago, I wrote about starting a new chapter.

I was leaving Banco John Deere and joining Rivio as a Senior Data Engineer, looking for a new challenge, a new environment, and most importantly, a chance to build something from scratch.

I ended that post with:

New repo, new problems, same obsession: building things people can trust.

Now, after some time in that new repo, I finally have some commits to share.

And the funny thing is, the biggest lesson wasn’t about GCP, lakehouses, or any specific technology.

It was learning what is actually worth building.

Starting from “something already works”

When I joined Rivio, there was already a simple data warehouse.

It used BigQuery, dbt Cloud, and Hex for BI. Data was being ingested from Supabase and MongoDB using Google Cloud Datastream.

It wasn’t a bad platform. Actually, it worked.

The problem was that the person who built it was no longer there, and the knowledge around the platform hadn’t really survived with it.

I spent my first weeks maintaining what already existed while trying to understand how everything worked, what the actual problems were, and where the platform needed to go.

At the same time, I had something that doesn’t happen very often.

I had the freedom to rebuild it.

There wasn’t a predefined architecture waiting for me. There wasn’t a long list of technologies I had to use.

So I started designing.

The first big decision, let’s build a lakehouse

I wanted to start with a lakehouse architecture.

It made sense to me to have a single architecture for data storage, processing, and analytics instead of having a separate data lake and data warehouse architecture.

I also had another reason.

I had spent years reading about these architectures, studying the tools, and discussing how they worked.

Now I had a chance to actually put some of those ideas into production.

That was a great learning opportunity.

I researched the GCP ecosystem, thought about the tradeoffs, and proposed an architecture that I believed would give us a good foundation.

And then I started building.

Some of the decisions I made back then are not part of the architecture anymore.

That’s actually one of the things I like most about this experience.

A good architecture isn’t something you design once and defend forever.

It’s something you change as you learn more about the problem.

The platform was not the real problem

One thing became clear pretty quickly.

The company needed a data platform, not because the existing platform was technically bad, but because we needed to become better at using data.

We are an AI company.

It didn’t make much sense to have a company built around data and intelligence while making it difficult for people to understand our own data.

At the beginning, the consumers were relatively small groups, mostly people from operations and commercial teams.

The use of Hex wasn’t very strong yet, so there wasn’t a huge business pressure around analytics.

But there was already a problem.

People needed data, but there wasn’t a clear way to get it.

Nobody really knew how to ingest a new source.

Sometimes nobody even knew which data should be ingested.

And without a platform, the natural outcome is predictable.

People start extracting data manually.

Different teams start having different numbers.

Someone creates a spreadsheet.

Someone else creates another spreadsheet.

Eventually, everybody has their own version of the truth.

At that point, the problem isn’t that you don’t have enough dashboards.

The problem is that you don’t know which number to trust.

Fortunately, we had the time to build the foundation before this became a bigger problem.

Today, having dashboards and reliable analysis is much more important to the day-to-day operation of the company.

Building the infrastructure

The first things I built weren’t actually data pipelines.

They were infrastructure.

The previous environment had been created manually, so one of my first priorities was to automate the deployment using Terraform.

Once the infrastructure was in place, I kept the existing ingestion from the two databases while rebuilding what happened after the data arrived.

Previously, CDC was landing directly in BigQuery.

I changed that approach.

The raw CDC events started landing in GCS first, preserving the raw data before processing it into Iceberg tables.

From there, I started building the rest of the platform around it.

I introduced data contracts to keep metadata in one place.

I removed dbt from the architecture and reused many of the existing dimensional models, migrating the transformations to Polars and DuckDB.

Composer became the orchestration layer.

And gradually, more data started flowing through the new platform.

One of the most interesting parts was that I was finally putting technologies I had studied before into production.

Iceberg.

Polars.

DuckDB.

Terraform.

GCP.

Kubernetes.

Data Contracts.

It was a great learning experience.

It was also where I started discovering that knowing how to build something and knowing whether you should build it are two very different skills.

The things I would do differently

This is probably the most valuable part of this experience.

Because if I rebuilt the platform today, I wouldn’t build exactly the same thing.

I would probably use more existing tools

One of the hardest parts was building the data processing layer ourselves.

Writing the transformations, creating the tables, handling the processing logic, and making everything work together took more effort than I initially expected.

Sometimes I wonder if using something more opinionated, like Dataform or another managed solution, would have been a better choice.

Building things yourself is fun.

But fun isn’t always a good architectural requirement.

I might not use Airflow

Composer gave us a powerful orchestration layer, but sometimes I wonder if we really needed it.

For some workloads, a Kubernetes CronJob or another simpler scheduling mechanism could have been enough.

This is something I keep learning over and over again.

Just because a tool can solve a problem doesn’t mean the problem deserves the tool.

I probably wouldn’t add DuckDB today

I originally wanted BigQuery to be our query engine, but not necessarily our processing engine.

That’s why having Polars and DuckDB available seemed interesting.

But in practice, I was the main person using them for transformations.

The flexibility looked great on paper, but the architecture was gaining complexity without having multiple consumers who actually needed that flexibility.

Today, I would probably simplify this part.

I was solving problems we didn’t have yet

This was probably my biggest lesson.

Early on, I was thinking about data lineage, data quality, governance, and all the other things a mature data platform should eventually have.

But we didn’t have enough data consumers yet.

We didn’t have enough use cases.

So I was spending time designing solutions for problems that didn’t exist yet.

That’s when I started thinking differently about platform engineering.

A platform shouldn’t be designed only around what it might become.

It should solve today’s problems while leaving enough room for tomorrow’s.

GCP after years of AWS

Another big part of this chapter was moving from AWS to GCP.

I had spent years working with AWS, so I expected the biggest challenge to be learning new services.

It wasn’t.

The services themselves are not that different.

What felt different was the platform experience around them.

At first, GCP felt like there was a wall between me and some of the things I wanted to do.

It was strange.

Maybe some of that was just my AWS bias after years of using it.

Today, things feel much more natural.

I still prefer AWS, probably because it is what I’m most comfortable with, but I can’t deny that GCP has a very strong ecosystem for data.

I also had to get back into Kubernetes.

I had worked with it before, back at QuintoAndar, but it had been a while.

It took some time to get back into it.

Apparently, Kubernetes doesn’t care how long you’ve been away.

It will still find a way to remind you that you forgot something.

What changed in me as an engineer

Technically, I learned a lot.

I got to put technologies into production that I had previously only studied or experimented with.

That was exactly one of the things I was looking for when I joined.

But the funny thing about learning more is that you don’t necessarily feel more complete.

You just discover more things you don’t know.

And that’s probably a good thing.

I also became much more conscious about cost.

The infrastructure team constantly challenged me to think about it, and that changed the way I looked at architectural decisions.

Before choosing a technology, I started asking not only:

Can we build this?

but also:

Do we need this?

How much will it cost?

What happens when the volume grows?

What complexity are we adding?

That mindset is probably more valuable than any individual technology I learned.

And then there was AI

One unexpected part of this chapter was AI.

Not just using AI to write code, but figuring out how it can become part of my daily engineering workflow and how it can interact with data.

That ended up being one of my biggest learning curves.

And I think that deserves its own commit.

Was it worth it?

Yes.

Absolutely.

I gained a type of experience that I don’t think I could have gotten by staying comfortable.

I got to build a platform from scratch.

I got to make architectural decisions.

I got to put new technologies into production.

I learned GCP.

I got deeper into infrastructure and Kubernetes.

I learned a lot about cost.

And I experienced what it actually means to work in a startup.

Startup life is a little crazy.

Things change quickly, sometimes without warning, and you need a certain level of resilience to not let every change get to you.

One thing I didn’t expect was to become one of the technical references in the room.

I thought I was joining a place where I would have more technical references around me to challenge me.

Instead, I found myself being that reference for other people.

That was uncomfortable sometimes.

But it was also exactly the kind of experience I needed.

Looking back

If I could go back to the person who wrote my previous post six months ago, I wouldn’t tell him to change the architecture.

I wouldn’t tell him which cloud to use.

I wouldn’t even tell him which technologies to learn.

I’d just tell him:

Go for it.

Not because every decision will be right.

Some won’t.

Not because everything will be easy.

It won’t.

But because you can’t learn everything from a comfortable position.

Sometimes you need to build something, realize you built too much, tear part of it down, and build it differently.

That’s experience too.

Six months ago, I was starting a new chapter.

Today, I’m still building it.

And there are still plenty of commits left.