The purpose of this blog is to record my experiences as I embark on a mission to further my career as a data engineer.

The Past

My background has been in the financial services industry. In each role I have had I have progressed further in the direction of data. Initially my skils where in quantitative analytics and Excel was the tool I used. However, it did not take too long to realise we were often treating Excel like a database and the sluggish performance was only deteriorating. Using Access was my first exposure to databases and I soon progressed to SQL Server. I have done work designing schemas, building ETL routines, building mutlidimensional cubes and implementing standard BI reporting layers. I have gained knowledge with each role, but have always been a power user from the business. As a result, my financial services domain knowledge is strong and my skills will technology are pretty strong but patchy.

The Present

I have just taken a consulting role at https://www.servian.com, where I will focus initially on data engineering work. This is a fantastic opportunity to broaden and harden my skill-set. I want to fill out my knowledge of competing technologies and the trade-offs clients should consider when designing a solution. I will get plenty of cloud engineering exposure and this is a very exciting part of the opportunity. I will use this blog to document my project work and hope I can use it to develop my writing and critical thinking processes. Perhaps someone else will find it interesting down the track.

The Future

Like many people with a passion for data, I find the world of AI and ML fascinating. My goal is be able to build scalable, efficient, fault-tolerant data pipelines that fit into an organination’s data science ecosystem. I am also very interested in data warehousing, which has been my core work in recent years. Developing an understanding of where data warehousing fits in the broader data framework of an organisation is a key goal. Data lakes are extremely popular at the moment, however I have a strong feeling the principles built up through 40+ years of hard experience in data warehousing will remain very valuable. In the same way that SQL is now reasserting itself as a key skill in this field, data modelling and data warehousing will also experience a resurgence.