Data Engineer Career Guide: Pipelines, Skills, and Interviews

Data engineering postings read like tool inventories: Spark, Airflow, Kafka, Snowflake, Databricks, dbt, and at least one cloud. Panels hire for something narrower: whether you can move data reliably, shape it so others can use it without a manual, and rerun a failed load without corrupting a table. What follows explains the work behind those keywords and how panels test it.

What data engineers spend the week on

The job is to get data from where it is produced to where it is used, in a shape and on a schedule people can depend on. Sources include application databases, app events, SaaS tools such as CRMs and payment gateways, and partner files; the destination is a warehouse or lakehouse that analysts, BI developers, data scientists and ML engineers query. Larger companies split the work between a central platform team and engineers embedded in domains such as payments or supply chain.

A realistic week for a domain data engineer might include:

• Fixing an overnight Airflow failure: a partner's daily file arrived with dates in a new format, so you patch the parser, rerun the affected partitions, and tell finance why their dashboard is a day behind.
• Replacing a nightly full dump of the orders database, which now takes four hours, with change data capture (CDC) that streams inserts, updates and deletes.
• Reviewing a dbt pull request that joins order lines to shipments at different grains and silently double-counts revenue.
• Tracing a jump in the warehouse bill to a dashboard that scans an unpartitioned events table every five minutes.

This role is most often confused with the analyst and BI roles that consume your tables. A simple test: if the question is why revenue fell last week, it belongs to an analyst. If it is why the revenue table landed four hours late, or why it counts some orders twice, it is yours. Backend engineers own the application database; you own its analytical copies, which is why schema changes and CDC come up so often. Where analytics engineers exist, they own the dbt layer between raw data and reports; elsewhere, you do.

Which skills employers name, and why each one matters

Panels weigh these tools very differently: the core group gets you through the door; the differentiators decide between two solid candidates.

Skill	Why it matters	Weight
SQL, including window functions and query plans	Every transformation, check and investigation runs through it; weak SQL sinks candidates whatever else is on the CV.	Core
Python	Ingestion code, PySpark jobs, Airflow DAGs, and their tests.	Core
Apache Spark	For data too large for one machine. Knowing partitions, shuffles and skew separates tuning a job from merely submitting it.	Core where volumes are large
Airflow or another orchestrator	Dependencies, retries, schedules and backfills. Azure Data Factory, Dagster and Prefect fill the same slot.	Core
One warehouse or lakehouse in depth	Snowflake, BigQuery, Redshift and Databricks store, price and optimise data differently. Depth in one transfers; a shallow list of four does not.	Core
dbt and dimensional modelling	Star schemas, slowly changing dimensions and tested models make tables usable outside your team.	Core in analytics-facing teams
Kafka and stream processing	Event pipelines, CDC, and anything that must be fresh within minutes.	Differentiator unless the team is real-time
Data quality, lineage and cost control	Freshness and volume checks, knowing which reports a change will break, and predictable compute spend.	Differentiator; expected from senior hires

Cloud choice shapes job titles in India. "Azure Data Engineer" postings usually mean Data Factory, Databricks and Synapse or Fabric; AWS roles name Glue, EMR, Redshift and Athena; Google Cloud roles name BigQuery, Dataflow and Cloud Composer, which is managed Airflow. The concepts carry across all three. The live data earlier in this guide shows which of them current postings in India name most often.

What each level is trusted with, and how titles vary

Experience	What you own	Common titles
0–2 years	Extending pipelines others designed, clean SQL and Python, tests, and debugging failed tasks from their logs	Associate Data Engineer, Data Engineer I, ETL Developer, or a generic engineer title on a data project
3–5 years	A domain's pipelines end to end: batch or streaming, incremental loads, safe backfills, code review	Data Engineer II, Senior Data Engineer
6+ years	Platform decisions (warehouse or lakehouse, table format, orchestration standards), data contracts, cost, mentoring	Lead, Staff or Principal Data Engineer, Data Architect, Engineering Manager

Above junior level, panels probe judgement more than syntax: why a MERGE rather than a partition overwrite, or why hourly rather than every five minutes. The kind of employer matters as much as seniority:

• Services companies run client projects, often migrations from on-premises Hadoop, Teradata or Informatica to a cloud stack. You rarely choose the tools, and certifications count for more because they help with client staffing.
• Product companies and startups expect you to own the platform, carry on-call, and feel the cost of every query.
• GCCs often build the parent company's global data platform. Governance, PII handling and access control take more of the week, and some meetings fall in overseas working hours.

How a data engineering hiring process usually runs

1. Recruiter screen: current stack, data volumes handled, notice period and expected CTC. Have a two-line answer to "what does your pipeline do, and how big is it?"
2. Online assessment: timed SQL, sometimes Python.
3. Live SQL and coding: window functions, deduplication, and flattening nested JSON in Python while you talk through edge cases.
4. Pipeline design: an end-to-end scenario on a whiteboard or shared document. It carries the most weight from mid-level up.
5. Tool depth: Spark internals, Airflow behaviour or warehouse specifics, chosen from your CV. Anything you list can be examined.
6. Hiring manager: an incident you handled, a trade-off you argued for, and how you deal with upstream teams who change things without warning.

Services firms often compress this into one or two technical rounds plus a managerial round, and may add a client interview once you are proposed for a specific account.

Questions that test real pipeline experience

These separate people who have run production pipelines from those who have followed tutorials; the edge cases you raise count as much as the answer.

1. Window functions. "Events arrive more than once; keep the latest row per event_id." Then: "Split each user's clicks into sessions, where a gap of over 30 minutes starts a new one." Expect ROW_NUMBER() partitioned by the key and ordered newest first, and LAG plus a running SUM of gap flags for the second.
2. Copy a mutable table. "Load a large Postgres orders table into the warehouse every hour. Rows are updated and occasionally deleted." Compare an updated_at watermark with log-based CDC, and say how deletes and new columns are handled.
3. Make reruns safe. "A daily job appends rows, fails halfway and is retried. What went wrong?" They want idempotency: overwrite the partition for the run's data interval, or MERGE on a key, so two runs produce the same table.
4. Backfill without breaking today. "Recompute a new column for two years of history." Cover chunking by partition, limiting concurrency, keeping daily loads running, and validating counts before swapping tables.
5. Skewed Spark job. "199 tasks finish in seconds and one runs for 40 minutes." That is skew on a join key: broadcast the small side, salt the hot key, or rely on adaptive query execution's skew-join handling.
6. Layout on object storage. "Store a year of events. Choose a file format and partitioning, and explain why partitioning by user_id is a mistake." Parquet for column pruning and compression, date partitions, and the small-files problem that high-cardinality partitions create.
7. History in a dimension. "Customers change city, but last March's revenue-by-city report must still match what was reported then." This is a Type 2 slowly changing dimension with surrogate keys and validity dates.
8. Late events in a stream. "Count orders per five-minute window, but some mobile events arrive an hour late." Separate event time from processing time, set a watermark that bounds how long state is kept, and decide what happens to anything later still: drop it, or correct the table in a daily batch.

Resume points that show scope, not just tools

Reviewers look for volume, freshness, reliability and cost, plus what you personally decided; "worked on ETL using Spark" says nothing. These examples use illustrative numbers, so substitute your own.

• Example: "Replaced a four-hour nightly export of the orders database with Debezium and Kafka CDC into Delta tables on S3, cutting data latency from a day to under 15 minutes and freeing the overnight load window."
• Example: "Rewrote 40 Airflow tasks to overwrite partitions by data interval instead of appending, which made a two-year backfill possible in four days with no manual clean-up."
• Example: "Moved dashboard queries onto date-partitioned, clustered BigQuery tables built by incremental dbt models, roughly halving monthly bytes scanned; added freshness tests that now catch upstream breaks before analysts do."

Common mistakes to fix:

• A skills block listing Hadoop, Hive, Pig, Sqoop, Spark, Flink, Storm and Kafka when your projects use three. Keep only what survives a tool-depth round.
• Calling a 15-minute batch job "real-time". Someone on the panel will ask.
• Hiding GUI ETL experience. Incremental loads and slowly changing dimensions in Informatica or SSIS are the same concepts; say so, and show you can express them in code.

Getting into data engineering and growing in it

A learning order that matches how interviews are weighted:

1. SQL until window functions, CTEs and execution plans feel routine.
2. Python for data work: paginated APIs, retries, and pytest tests.
3. Dimensional modelling from Kimball's The Data Warehouse Toolkit, then dbt to practise it with tests.
4. Airflow running locally in Docker, scheduling that dbt project with retries and a backfill.
5. Spark on a dataset larger than your laptop's memory, then Kafka basics.

Projects that show production thinking rather than a tutorial:

• An idempotent daily pipeline from a public API: partitioned Parquet, dbt models with tests, and a README showing that rerunning a past date produces identical output.
• A CDC demo: Postgres to Debezium to Kafka to a lakehouse table updated with MERGE, handling updates, deletes, and a column added upstream mid-stream.
• A cost write-up: take a slow query on a large public dataset, measure what it scans, fix it with partitioning and clustering, and report before and after.

Backend developers usually need to add modelling and analytical SQL; ETL tool developers need Python, Git and a code-first orchestrator; analysts who enjoy the dbt side often cross over through analytics engineering. From here, people grow towards platform and architecture roles, or sideways into ML engineering, where feature pipelines are data engineering under another name. If the consuming end interests you more, compare the BI developer and data scientist guides.

Find openings and keep preparing

Current listings: Bengaluru, Hyderabad, Pune, India nationwide and remote. Before you apply:

• Note which cloud and warehouse each posting is built around, and lead your CV with matching evidence; How to Read a Software Job Description helps you sort hard requirements from a wish list.
• Adapt the four-week preparation plan, giving one week to SQL window functions and one to talking through pipeline designs out loud.