My post on which languages to learn as a Power BI developer resonated with many people so I thought I’d take a step back and write about how to navigate the various facets of the business intelligence job market. Where language selection is effectively a tactical tool choice, in this entry we’ll focus on the strategic framing within which you can make career orientation decisions. Both entries are meant to be read together.

A primary concern for any professional is to maintain a relevant skillset throughout their career, and that’s an even more pressing in the tech sector at large, and for data professionals in practicular. I aim to bring clarify to hybrid skillsets and ambiguous job titles that seem to mean different things to different people.

1. Multi-Specialists Working at Intersections Are Vital but Hard to Pin Down – The Case of the Sappers

Allow me to start with a sidebar on military organization and family history. Feel free to skip this intro but please indulge me, I promise it will make sense.

There’s a combat engineering tradition in my family, as my grandfather and father were Colonels in that Army specialty, and while I shunned the pursuit of a military career, I did ten months of military service in combat engineering and am a Reserve Lieutenant.

My granddad built an airport in a rice paddy with the Foreign Legion during the Indochina war in 1952, and my dad made tanks cross the Rhine river during Cold War training maneuvers, back when France had troops stationed in Speyer Am Rhein in what was known as West Germany at the time. I didn’t do any active-duty work, but I did learn how to blow up a bridge! I grew up hearing about military matters – you can’t escape it when you’re an Army brat – and that left an impression.

The captain who taught me during my military service served within Foreign Legion troops deployed in Lebanon in the early 80s and had an interesting perspective on doctrinal purity vs. practical effectiveness. Here’s what one of his NCOs reportedly used to say about how to weight explosives to blow up a bridge by the size of the truck transporting them:

“Small bridge, small truck

Big bridge, big truck”

Smartass NCO – inefficient maybe, but effective! Possibly apocryphal, still, I’ll always remember it. In the end, do the job!

Wait, what’s the deal with blowing stuff up, you might ask? The purpose of combat engineering is to clear the way for your troops and obstruct the way for the enemy, in other words, to provide and deny mobility. That entails a wide variety of specialty skills such as:

• Crossing rivers with mobile bridges and destroying constructed bridges
• Clearing and creating obstacles, including laying and removing landmines
• Building and destroying base infrastructure

Combat engineers are thus the first to go to a contested area as you don’t want to send infantry and cavalry through a landmine field. And they’re the last to leave as they clean things up, sometimes for years after a conflict is over to dig out unexploded ordnance and mines. There are even subspecialties involving trains or air force support.

A Combat Engineer is more of a technician than most other ground troops but is still a soldier as he might well end up in combat situations. Depending on the country and era you may find pioneers either in their own units – France for one used to have a bunch of Génie regiments – or more commonly embedded within larger Army or Air Force units. But they’re odd ducks that are hard to pin down, as is reflected by their weird vehicles that are literally hybrid tank/crane/dozers and boat/bridges.

So what’s the relevance of this long-winded intro? What to make of sappers within the military sounds similar to asking whether BI should be within IT or report to the CFO or be embedded withing various divisions (sales BI, HR BI, marketing BI etc.). My take is that here’s no right answer, it’s entirely contextual to the organization’s industry, size, and culture. Business intelligence is the combat engineering of the corporate world, it’s a hybrid job meant to solve problems for other people. A sapper is a soldier and an engineer. A BI professional is a business analyst and an IT technician.

However, there are infrastructure-focused sappers (more engineer than soldier, like my grandfather was) and combat-focused sappers (the opposite, like my father was), just like there are IT-first and business-first BI jobs. Does it really matter? To transition to our civilian concerns, I would fret less about who is nominally my boss, and more about whether they understand what my job is about and give me the right resources and support. But as a BI professional we’re about to see that you do want to determine whether your primary focus is on the engineering side or the business side, it’s unlikely to ever be a perfect 50-50 balance.

2. Picking Primary and Supporting Skills with Intent

Let’s move on to another analogy where combat is now waged virtually and for fun. Again, bear with me, this is all relevant. Many role-playing games and other game archetypes using character classes offer the player the ability to combine two or more classes, leading countless kids to fantasize about building the ultimate paladin wizard who can prevail with brains and brawn alike. But because skill points are limited, this is more often than not the best way to gimp your character by making it a master of none, underpowered across the board jack of all trades.

However, well designed games do offer ways to create viable multi-talented character “builds”, usually through a combination of skill and equipment choices, provided the player shows restraint and acumen in picking complementary skills and combining them with just the right balance and optimal synergies. This can be very rewarding and fun when skill combos unlock big rewards. In the real world, you’ll see this in people picking smart combinations of university major and minor.

In my entry on BI languages, I argued that hyper specialization had limited appeal, significant drawbacks, and a lack of real-world relevance. So I do think that BI career tracks will require you to multi class almost by design. But that doesn’t mean you can just spread around your skill points randomly – putting two points in Python, one in color theory, five in financial concepts, four in data modeling, and one in internet security, just because they sound cool.

Instead, it’s best to build up your skillset with intent and alignment with your purpose, along these dimensions:

• Learning order: some skills are fundamental and underpin everything else you’ll be doing, so they need to be acquired first. If you don’t know how to gather requirements or model data, you’ll be stuck to being a glorified data entry clerk.
• Internal coherence: some skills need to be combined for best effect. If you don’t understand data modeling, it’s hard to make the right ETL choices.
• External alignment: what’s your role within your team? Are you tanking on the front line or healing from a distance? If you’re primarily supporting other analysts by massaging data in the warehouse, you probably don’t need to go deep into visual design skills.
• Versatility: are you always solving the same kind of issues in the same settings, or do you frequently need to jump into different roles? How much of a Swiss Army knife you need to be depends on whether you work in one company or as a consultant, and how big your team and organizations are. Even a pure damage dealing class might be on crowd control duty once in a while, and even within damage dealing, sometimes you do it on a single target, sometimes on an entire area, sometimes you’re shooting at a static target, sometimes it’s moving… you won’t get away for long knowing only how to do a single narrow thing.

I can’t harp about this enough, these decisions are all about context, don’t seek absolute answers.

3. How to Choose Your Major: Dealing with People, Processes, Tools, and Culture

Organizations operate based on a combination of people, processes, and tools. Which one of those three dimensions you’re most comfortable with should help you determine what type of BI professional you’d like to be:

If you’re comfortable dealing with the ambiguity, mood swings, occasional politics, but also the energy and creativity that flows from interacting with other humans, then you’re more likely to succeed on the business side, from gathering requirements to building visualizations to coaching other people. I call that role “CFO whisperer.”

At your best you may know better than your clients/users what it is they need, which is a very rewarding feeling when things just click. It’s even more rewarding when users realize this happened and credit you for it! One of my clients recently told me verbatim that I was able to articulate better than him his own use case for the purpose of hiring and briefing a developer. That’s because being bilingual business/tech puts you in the critical role of interpreter, in the absence of which people from different departments often talk past each other.

You may also in time become an executive yourself – VP of sales, CFO, CEO etc. – with a strong minor in data crunching, like many are.

If you like processes and organization, then the project management, governance, and administrative side may be a better fit. Not necessarily the most fun part, but the BI trains do need to run on time. The reality of IT projects is that they are just looking for an excuse to fail. Goalkeeping is an important part of avoiding failure, if not guaranteeing success.

If tools and software are your favorite thing, then data engineering and other back-end duties such as data security will appeal to you. This may or may not involve actual software development, as no/low-code data tools are pretty pervasive, especially if you operate in a Microsoft shop. The “Modern Data Stack” favored by many start-ups tend to be more code-heavy and closer to software engineering DevOps practices with source control, Continuous Integration/Deployment, and so forth.

All that being said, you can’t max out in a single dimension and expect to succeed. Organizations have smartened up to the fact that brilliant assholes might be spectacular individual contributors but so toxic to the team and culture that one shouldn’t put up with their drama. So you can’t be a 10 in tech and/or process and a 0 in human. On the other hand, if you’re maxing out on empathy and soft skills without any technical knowledge or any regard for process, there might be roles where you can succeed, but BI is definitely not it.

I do find that younger generations tend to need to be coddled a bit too much, and many HR departments have been indulging them, with the media-driven obsession of accommodating Millennials and their successors. There’s such a thing as being too soft and self-centered, what with that tendency of portraying every first-world minor inconvenience as a “mental health” issue of high importance. Maybe read about the life of people enduring wars and spend less time navel gazing?

But we shouldn’t be longing for the gung-ho, hyper-aggressive culture that companies like Microsoft or Oracle had back in the 90s. Can you imagine company meetings where high-level execs ask you to “nail the coffin” on a competitor? Healthy organizations should aim for a balance between performance and empathic concern for genuine human needs. You can be driven without steamrolling your colleagues!

4. Core Business Intelligence Roles

Most BI projects follow a similar structure that starts with defining what the business end goal is, mapping how to get there in light of data sources and software used by the organization, getting and shaping said data, turning it into visualizations, deploying reports to end users, and most likely iterating through these steps as a series of cycles. This thus leads to roles that may or may not be handled by separate individuals depending on project size and complexity:

• Business analyst
• Software architect
• Data engineer
• Report designer
• Administrator
• Project manager

Job titles vary and new phrasings emerge all the time, such as the “analytics engineer” title mostly pushed by dbt. Industry participants love to fret on Reddit and the likes about what these semantic tweaks may mean, which can sometimes be useful but is too often existentialist thumb twiddling.

Greg Deckler covers these roles at length as well as how to approach BI career development in his Learn Power BI [Amazon affiliate link] book. I’ll give my own take below.

4.1. Business Analysis: The Key to the Why

In this category you’ll find a variety of roles and titles including data analyst, BI analyst, business analyst, or even functional consultant. Whether you’re applying to a full-time job or a consulting engagement, it’s important to dig below the title to find out what the recruiting organization really means: what are you expected to produce, who are your (internal) customers, how will you be supported, and what you’ll be evaluated on. Maybe the focus is upstream on gathering and clarifying requirements, which is a function necessary in any technology project, BI or not. Or maybe you’re building reports all day long based on someone else’s requirements.

In any case to be effective in business-first positions, you need to develop functional and industry domain knowledge. There’s not a single BI project I’ve delivered where I haven’t used my background in business operations and my business school education. You’re not supposed to be as deeply knowledgeable as, say, a CFO with 20 years of experience in a very specific industry, but you need to know how to ask the right questions and drink from the firehose to quickly know not just what your users want you to build, but more importantly why they need it and how they’re going to use it to conduct business.

That way you’ll even be able to deliver more than what they say they want, but what they actually need. If you’re working with commodity traders you may need to understand what contango and backwardation mean, while a CFO might ask you to calculate Days Sales Outstanding, just to name two random examples. In some cases you’ll actually be the one explaining KPIs to your users. You’d be surprised how many small business owners get confused by the difference between markup and gross margin.

You can be more valuable and successful in this area by developing your knowledge of systems of record in software categories such as ERP, CRM, or project management, because this is where most of your data will be coming from. Understanding the key entities, workflows, and processes where all this data is coming from will help you both to define metrics and pursue the inevitable data hygiene initiatives that will need to address data quality issues you’ll no doubt run into. Building a report out of, say, Salesforce, means you’ll have to understand how data flows from Leads to Accounts and Contacts to Proposals to Quotes to Deals. And be practical, don’t stay in the abstract. I always recommend that analysts have at least some exposure to the user interface of key source software.

4.2. Data Engineering: The Essential Plumbing That Consumes Most of the Resources

Data engineering is in charge or extracting, transforming, and loading data from sources to destinations (aka sinks), via patterns commonly known as:

• ETL (Extract, Transform, Load) – the data schema is established on write as you load data in its destination after transforming it. This is typically associated with data warehouses and implies a serious data modeling effort.
• EL(T) – the data schema is established on read, meaning you load the data as is and transform it later, which is associated with data lakes and lakehouses. Data modeling may or may not be performed at some later stage.
• Batch vs. Streaming – toolsets for discrete ingestion after the fact in batch mode vs. continuous streams are different for the most part, so handling data streams is effectively a skill subset in its own right.

You may also hear about “data wrangling” or “data shaping”, they’re all pretty much synonyms. Functionally it doesn’t matter whether you’re setting up a data pipeline by clicking on a GUI or writing code, as long as you eventually land data where you need it to be, in the state you need it to be. But some people call the former “ETL developer”, which they’d use for someone who uses GUI tools such as Alteryx, Talend, Pentaho, or ADF, while they’d reserve the term “data engineer” to people who manually write code.

While it’s currently pervasive, the phrase “data engineering” is actually fairly recent and is taking inspiration from modern software engineering. It is thus leaning towards the code side, as well as modern DevOps practices such as using source control and automated testing for Continuous Integration/Continuous Delivery (CI/CD).

To coordinate discrete ETL tasks you’ll find the concept of orchestration, as data pipelines often need to be executed in a sequence of dependencies. A senior data engineer lives in that realm, with a good sense of the entire lineage of data across pipelines and all the dependencies and potential failure points along the way.

Real world data is never 100% clean, thorough, up-to-date, free of ambiguity, or ready to be integrated with other sources. Data engineering is thus an essential skill without which only the most trivial of BI projects can be delivered. It commands some of the highest salaries in the BI space, is stubbornly resistant to full automation, and will probably never go completely out of style. In my opinion you won’t go very far in BI without having a modicum of data engineering skills and proficiency with at least one GUI tool and/or ETL language. You might table orchestration concerns until a few years into your career, but any project that involves more than a couple of sources will likely require going beyond simple, isolated ETL tasks.

4.3. Data Modeling: Giving Data Its Analysis-Ready Shape and Meaning

Data Modeling is closely ties to data engineering, as it determines the shape of, and relationships between, your data tables. People have been going back and forth for 30 years between enterprise data warehouse architectures from the like of Bill Inmon or Ralph Kimball versus using wider tables that mix facts and dimensional attributes. Taking that latter approach needs to be a conscious decision, meaning that you need to understand modeling even if you’re not going to do much of it.

Lately even people using Tableau – a tool that historically was all about joining everything in wide “frankentables” – have been warming up to dimensional modeling, the approach that is universally recommended in the Power BI world. Does everyone need to know the subtle distinctions between types of Slowly Changing Dimensions or how to manage surrogate keys? Maybe not. Or do you need to know how to set up OLAP cubes? Again, it depends on your environment, but precomputed cubes are no longer the must-have tech layer they used to be. On the other hand, I’d venture that you can’t call yourself a BI professional if you don’t at least know star schema basics.

Don’t lose sight of what data modeling is ultimately about: whatever the shape and structure of your collection of tables, you’ll need to calculate the metrics required by your clients and guarantee they’re accurate. Domain knowledge and integration concerns are not going anywhere, regardless of technical implementation choices.

Some organizations create a distinction between data engineers importing data from outside sources into the analytical stack while analytics engineers to subsequent transformations within the analytical realm, typically to create “gold” datasets ready for analysis and visualization. In this case, the former role is more focused on orchestration and automation, while the latter would typically need to align data with business requirements via data modeling. Again, your mileage may vary in terms of who does what and what their job title is.

If you’re a leader, then all the skill mapping we’ve been talking about at the individual level applies to your team composition and dynamics.

4.4. Visualization: A Minor That Deserves Respect, But It’s Not “The Job”

People who don’t know data might think BI is mostly a visual design job, as if you were drawing shapes in Photoshop or PowerPoint. Software vendors contributed to that misunderstanding because they want everything to sound braindead easy. BI professionals obviously know better: the visual part is just the top of the iceberg.

Don’t get me wrong, we do need to pay attention to visual design, from gestalt principles to color theory, white space, composition, and more. I do see many reports out there that are, quite frankly, very poorly designed and deliver such a dismal user experience that nobody uses them. And beyond the strictly visual part, paying due attention to accessibility concerns is also part of the job.

On the other hand, don’t let contests organized by vendors fool you. The purely visual part is 10% of the job, nobody is a “BI report designer” who can get away with never doing any ETL, modeling, or admin. In the corporate world, people don’t have time to engage with fancy visuals that require a long explanation, and if you over-visualize it, they’ll be quick to ask for plain tables, or worse, an Excel data dump.

4.5. Administration, Governance, Adoption, People Leadership: Making Sure It Sticks

And now the fun part: dealing with people to get them to use your BI solutions, and do so responsibly. At a junior level you might be stuck to doing administrative tasks like managing secure access and users. As you grow more senior, non-technical BI roles shift to handling organizational and even cultural concerns. Your organization should not neglect these responsibilities if they don’t want to leak confidential data, end up with an unmanageable hell, or a BI desert that nobody cares about. As a BI professional, if you don’t bear these responsibilities yourself, you’d better know who does in your organization and work closely with them, otherwise your pristine data pipelines and brilliant dashboards won’t see much production use.

More broadly, a few years into their careers, most people will face a fork where they have to choose whether they want to keep working as an individual contributor or start managing teams of other people. Excelling as a data engineer doesn’t mean you’ll be good at managing a data engineering team, let alone an entire analytics department, just like being a quota-shattering sales rep doesn’t mean you’ll enjoy or perform as a VP of Sales.

There’s no right answer here. Go back to your self-assessment – hopefully coherent with how others perceive you – of where you fit in the people-tools-processes triad. Avoid pursuing a path that sounds more prestigious if it’s going to make you feel miserable. There are also different ways to work with other people as a senior. One path is hierarchical, where you’re responsible for hiring, firing, evaluating and so forth. Another path doesn’t have these clear responsibilities and instead is more focused on elevating other people’s skills through mentoring, which can be very rewarding in its own right.

4.6. Architecture: Putting Everything Together (Indoor Sunglasses Optional)

All these moving parts are necessary, but someone needs to make them fit together. This job is commonly known as Software or Systems Architect, a fairly senior role that combines good technical fundamentals across the OSI model with understanding of human organization and dynamics.

The Architect gives everyone a common sense of direction while making sure that software components complement each other well without leaving big functional gaps or obviously failing to meet functional requirements. That latter part is hard, especially in terms of scaling, as the proof is in the pudding.

If you like to go back and forth between the strategic big picture and tactical implementation details and have a fetish for flowcharts, then this might be the job for you. Did I mention I love flowcharts?

4.7. What About Big Data, Data Science, Machine Learning, Artificial Intelligence?

I’ll make this short as I’m irritated by the confusion fueled by clickbait media, clueless HR departments, and grifting bootcamps, colluding to give the impression that somehow everyone is going to make six figures overnight working from home by becoming a “data scientist” working with “big data”, whatever that means. The “data scientist” title was created to make a data analysis job sound more appealing! The truth is, technology has always been subject to hype/bust cycles, so always look at the breathless pronouncements with a grain of salt, at least in the short term.

Is there a place to develop machine learning algorithms and apply statistical methods to predict the future by using humongous amounts of data? Sure. Is that place your department? Probably not. Most companies, even those selling all these fancy new technologies, are still struggling with more prosaic business intelligence challenges. I’m very much in the “walk before you run” camp. Moreover, BI and DS/ML are adjacent and complementary, they’re not the same.

Coming Soon – Axes of Growth: How to Plan & Structure Your Career

Anything or anyone that stagnates will sure enough start to shrivel and die. To keep your work trajectory on a healthy track, you need to ask yourself whether you keep learning things that will remain relevant, sustain your interest, and hopefully, get you more money as well. If you’re “quiet quitting”, you’re not being smart, you’re quietly quitting on future you.

Stay tuned for a follow-up entry that will cover things to look for in your current or next job that will serve as building blocks for your evolution further down the road.

Related Reading

• Which Languages Should You Learn as a Power BI Developer?
• Third-Party Tools to Ease Power BI Development and Increase Analyst Productivity