Database and operations infrastructure for the billing system

Seventeen years around the mobile operator in Düsseldorf, in several assignments with changing clients: from 2002 to 2007 directly for Vodafone; from 2007 to 2017 for the outsourcing providers EDS, HP Enterprise Services, Hewlett Packard Enterprise and DXC Technology, who had taken over software development for Vodafone and were our clients in that period; from 2017 to 2019 directly for Vodafone again. For many years the focus was the Amdocs-based billing system with its satellite systems, one of the largest Oracle landscapes in German mobile telecommunications.

What we implemented

From 2002 to 2007 we led the DBA group for the test environment of the billing system: around 1,600 test environments with about 120 databases and more than 5.5 terabytes, plus support for the production DBAs with 38 instances and over 35 terabytes. In that time we migrated from Oracle 8i to 9i, switched from the rule-based to the cost-based optimizer, designed a failover cluster concept with Veritas Cluster Server and EMC SAN, introduced Oracle Enterprise Manager Grid Control and co-led the development of an infrastructure support software of around 20 person-years.

From 2007 on the respective outsourcing provider was our client. For EDS, HP Enterprise Services, HPE and DXC Technology we took on project and sub-project management: the Oracle 10g upgrade of the entire landscape, the offshoring of infrastructure support for the test environments to Amdocs in India, the upgrade from Solaris 8 to 10 across 19 Sun servers with 1.2 terabytes of RAM and 35 terabytes of disk space, the introduction of mediation and fraud management systems and the migration of application development to a new data center with VMware, Oracle RAC and Nagios monitoring. Then the operation of the development data center with Linux, Solaris, HP-UX and AIX, two DNS domain migrations and platform moves from Solaris to RHEL. For performance reasons we also programmed a migration from MySQL to Oracle, schema and data included.

Most recently, from 2017 to 2019, we built the new production environment of the online sales platform as DevOps engineer: Apache, HAProxy and Varnish, MySQL Percona, RabbitMQ, ELK, Docker, Jenkins, Ansible and Terraform, plus an assessment of AWS for test and development environments.

Result

A billing system that ran across several Oracle and Solaris generations, a data center that moved without losing its applications, and a production platform that stayed in operation after launch. And an environment in which we were in demand for seventeen years, at Vodafone itself as well as at its outsourcing providers.