In the rapidly evolving world of software development, the emergence of platform engineering has marked a significant shift in how organizations approach software delivery and operations. This blog post delves into the three emerging categories within this movement: Internal Developer Portals, Internal Developer Platforms, and the next generation of Platform as a Service (PaaS 2.0). We'll explore the journey that led us here, the nuances of these platforms, and the critical differences between an Internal Developer Platform (IDP) and a PaaS.
The Emergence of key Platform Engineering Categories:
- Internal Developer Portal: This is the gateway through which developers access tools, scripts, dashboards, and frameworks designed to streamline and manage the development process.
- Internal Developer Platform: These platforms are more comprehensive, offering an integrated set of tools and services for building, deploying, and managing applications.
- Platform as a Service (PaaS 2.0): The next evolution of PaaS, offering enhanced capabilities and greater flexibility, tailored to modern development needs. Some of these also offer to run “in your own cloud.”
The journey to the current state of platform engineering is paved with various developments:
- Internal Tooling: Initially, DevOps teams created scripts, dashboards, and frameworks to manage environments and configurations. This was the foundation of today's more sophisticated platforms.
- Infrastructure as Code (IaC) Tools: The success of IaC tools like Terraform, Pulumi, and Crossplane has been instrumental in shaping platform engineering, allowing for more efficient and scalable infrastructure management.
- Originally, the idea was that these tools enabled developer self-service directly, since they were just code or configuration, vs. the previous generation of ops-centric tooling such as Chef, Puppet, Ansible, SaltStack
- In practice, they still ended up with devs throwing work “over the wall” and did not achieve the “shift left” goals they set out to.
- Kubernetes (k8s) Ecosystem: The widespread adoption of Kubernetes as a deployment layer atop cloud infrastructure has led to a rich ecosystem of dashboards and plugins, further advancing platform capabilities.
- Risk and Compliance Considerations: With the shift towards in-cluster options over cloud vendor-managed services, organizations face higher maintenance costs and compliance burdens, despite potential cost savings.
- Real-World Application Challenges: Tools like Backstage, which build on Kubernetes, promise self-service but often face challenges in configuration for real-world applications.
- Developer Environment Evolution: The move towards microservice-oriented architectures has led to the development of shared clusters for development and remote tooling like Tilt, Telepresence, and Skaffold, enhancing integration testing and collaboration.
- Platform as a Service - Heroku (and its generation of companies) showed the world what is possible from a developer experience perspective, encouraging internal teams to push their own tooling in this direction. On top of the stack of tools described above, one direction for next-gen PaaS is to build “internal platforms” that achieve the same developer experience on top of private tools. Another direction has been “Heroku 2.0” where more advanced and capable, or more language-specific or language-integrated tools have emerged. Vercel, Railway, Render, Fly are some examples here.
While both IDPs and next-gen PaaS offer self-service capabilities, their approach to multi-cloud support, customizability, and support for various workloads and SDLC phases significantly differ. Key areas to explore when looking at these tools include:
- Customizability: Unlike traditional PaaS, modern platforms offer more control over resources, allowing for configurations like sidecars alongside service workloads. For example, Coherence offers custom helm support for deploying any service definition you need!
- Support for Non-Web-Facing Workloads: This includes worker queues, cron jobs, and internal microservices. Also consider the cloud resources needed to support these workloads, such as cron scheduler configuration, task queue services (e.g. SQS on AWS), S3 buckets, and many more.
- Advanced CI/CD Support: Internal developer platforms provide robust support for testing, integration testing, and data management (seeding or data snapshotting) in ephemeral environments. They also consider the developer experience of “release management” and how code will move from less-restricted to more-critical environments in different ways (consider staging/UAT/production as simple examples of these).
- Security and Compliance: Features like container vulnerability scanning and other automated pre-release safeguards are crucial in modern platform engineering.
- SDLC Phase Support: Comprehensive support for different phases of the software development lifecycle, from development to pre-production and also production, is a key feature of these platforms.
- Day 2 Operations: Modern platforms offer tools for database access, REPL, and role-based access control (RBAC) for various resources.
- Tool Integrations: Integration with a wide range of tools, beyond just version control systems like Git, is essential for modern platform engineering.
- Ability to live alongside, and even manage, custom or 3rd party infrastructure: A platform can automate and manage parts of your infra stack, while you should still be able to use other tools ranging from IaC to ClickOps to multiple platforms, as it makes sense for your team and applications.
- At Coherence, we have many examples of terraform/pulumi, or other databases offered as a service (Snowflake or MongoDB for example) living alongside the infra we manage, and integrating very seamlessly. We see this as an important part of the platform vs. PaaS mindset!
- There are also various lifecycle hooks, including our github actions integration and our AWS EventBridge features to directly manage infra based on Coherence events. Lots more to come in the new year on this front, based on continued feedback from customers and prospects. We’d love to hear your thoughts here as well.
The platform engineering landscape is continuously evolving, with a range of options that offer unique benefits and challenges. Understanding the nuances of an Internal Developer Platform is crucial for organizations looking to optimize their software development and operational processes.
At Coherence, more than anything else, we see teams still stuck in the mentality of feeling that they need to build their own platforms by integrating Infra as Code, custom dashboards/portals, and CI/CD tooling. This happens with the best of intentions - it’s because the needs outlined above are complex and simple “out of the box” solutions don’t meet them all. We hope that as the power of a true platform becomes more well-known, the “build vs. buy” arguments will change into “what do I need to buy” - and that teams will focus on velocity over re-inventing the wheel! The right mentality here, in our view, is to look for tools that take the simple work off your plate, deliver value across different phases of the SDLC, and integrate into the rest of your tools in flexible ways.
