Modernizing 60+ GCP Cloud Functions: Gen1 to Gen2, builds halved
Every Cloud Functions pipeline was a little different and most were slow. I standardized all of them, cut checkout time by about 80% and build time in half, and laid down a safe path from Gen1 to Gen2.
A staged migration; production follow-up also checked the runtime service account.
Contribution at a glance
- My responsibility
- Function-pipeline standardization, runtime migration and identity follow-up.
- Scope
- Phased development/staging delivery and distinct later production operation.
- Collaboration & boundaries
- I worked on the infrastructure and deploy path; a planning epic is not proof that every production function completed migration.
- Outcome
- Standardized deployment work and a production follow-up that pinned runtime identity after an implicit service-account change.
A fleet of Cloud Functions tends to grow one function at a time, and the pipelines grow the same way: copied, tweaked, and never unified. The result was dozens of functions across several repositories, each deploying through a slightly different Jenkins pipeline, most of them slow, and none of them ready for Gen2. I standardized the whole set and modernized the runtimes, sanitized here to the technique.
Delivery scope
This was a phased migration and pipeline-standardization effort. DEV/STG work and later production operation were distinct stages. A production follow-up found an implicit runtime service-account change and pinned the deployment identity to prevent recurrence. The planning epic alone does not establish the state of every production function.
Where the time was going
Two things dominated every build. First, the checkout: each build did a full clone of the deploy repository, 30 to 60 seconds of pure waiting before any real work. Second, dependencies: nothing was cached, so every build reinstalled everything from scratch. Multiply that across dozens of functions deploying many times a day and it adds up to real engineer-hours and real cloud build minutes.
What I changed
- Clone-or-update checkout. Instead of a full clone every build, the pipeline fetches and resets when the repository is already present. Checkout dropped from 30 to 60 seconds down to 5 to 10.
- Dependency caching. Builds now reuse cached dependencies instead of reinstalling, taking cache reuse from zero to 80 to 90%.
- Smaller deploy packages. Trimmed what gets shipped, roughly halving the deployment package size.
- Runtime updates. Brought Go, Node.js and Python to current, supported runtimes on Gen1, and added the Gen2 equivalents.
- Gen2 pipelines and examples. New Gen2 deploy pipelines plus working examples in all three languages, so adding a Gen2 function is copy-and-go, not research.
Along the way I fixed a batch of long-standing papercuts: an incorrect workspace path, a vendoring inconsistency from blank imports, an entry-point mismatch between Gen1 and Gen2, and oversized deployment packages.
Before and after
| Metric | Before | After | Change |
|---|---|---|---|
| Checkout time | 30 to 60 s | 5 to 10 s | ~80% faster |
| Total build time | 3 to 5 min | 1.5 to 2.5 min | ~50% faster |
| Deployment size | 100% | 40 to 50% | ~50% smaller |
| Cache reuse | 0% | 80 to 90% | new |
Why Gen2, and how to get there safely
Gen2 is not just a version bump. It runs on Cloud Run under the hood, which brings concurrency (up to 1000 simultaneous requests per instance instead of one), native Eventarc integration, faster cold starts, and more granular pay-per-use. The migration is where people get nervous, so I kept it boring:
- Start in dev with HTTP functions on the updated runtimes, and validate each one on its own.
- Move to Pub/Sub-triggered functions only after the HTTP ones are proven.
- Use blue-green cutovers so traffic shifts only once the new version is healthy.
- Replicate through staging, then production, starting with non-critical functions before the critical ones.
What I would tell someone starting this
- Standardize the pipeline before you optimize it. You cannot speed up twenty different pipelines; you can speed up one and apply it twenty times.
- The cheapest build win is usually the checkout. Stop full-cloning on every run.
- Cache dependencies deliberately. Zero to 80% cache reuse is often the single biggest drop in build time.
- Migrate runtimes in a fixed order (dev, staging, production; HTTP before Pub/Sub; non-critical before critical) and the scary migration becomes routine.
Decisions and next steps
A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.
Alternative and rationale
Per-function pipelines offer local flexibility; a common deploy path reduces repetition but must preserve runtime and trigger differences.
Trade-off
New runtime generations can change identity and logging as well as compute. Deployment success alone is insufficient acceptance.
What I would improve next
Next improvement: maintain a per-function migration inventory with trigger, identity, logging and rollback acceptance checks.