KB: Airflow Operators

In Apache Airflow, operators are tasks that perform specific actions in a workflow or Directed Acyclic Graph (DAG). Each operator defines a particular action or job, such as running a script, transferring files, or interacting with external systems. Airflow provides various types of operators, which can be categorized broadly as follows:


1. Basic Operators

  • PythonOperator: Executes Python functions directly.
  • BashOperator: Runs Bash commands.
  • DummyOperator: Used as a placeholder or for creating dependencies.

2. Transfer Operators

  • Used to transfer data between different systems or databases.
  • Examples: S3ToRedshiftOperator (transfers data from S3 to Redshift), MySqlToS3Operator, S3ToSnowflakeOperator.

3. Sensors

  • Special operators that wait for a specific condition to be true.
  • Examples: FileSensor (waits for a file to be present), ExternalTaskSensor (waits for another task to complete), TimeDeltaSensor (waits for a specified time).

4. Hooks

  • Not exactly operators, but hooks allow operators to connect to external systems (e.g., databases, APIs).
  • Commonly used with custom operators to extend functionality.

5. Custom Operators

  • You can create custom operators by subclassing BaseOperator, which is helpful when standard operators don’t meet specific requirements.

6. ETL Operators

  • Specific operators for ETL processes, like BigQueryOperator or HiveOperator.

7. Utility Operators

  • Used for more advanced workflow control.
  • BranchPythonOperator: Chooses which path to take in the DAG.
  • SubDagOperator: Allows running a DAG as part of another DAG.

Each operator has its own set of parameters, allowing customization, and can be chained together to define complex workflows. These are essential for building workflows that manage data pipelines and automate tasks effectively.


 Airflow Operators: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html


Comments

  1. python.dataSeptember 27, 2026 at 11:00 PM

    ### Airflow Operators

    **Airflow Operators** are the building blocks of an Apache Airflow workflow. An operator defines **what task needs to be performed**, such as running Python code, executing a Bash command, transferring files, or running a SQL query. Each task in an Airflow DAG is usually created using an operator. Common examples include **PythonOperator** for executing Python functions, **BashOperator** for running terminal commands, **SQL operators** for executing database queries, and **EmailOperator** for sending emails. Airflow Course.Operators make data pipelines easier to organize, automate, schedule, and monitor because Airflow controls when each task starts and how tasks depend on one another.

    ReplyDelete

Post a Comment

Popular posts from this blog

Electron Process Execution Failure with FSLogix

  Technical Note: Electron Process Execution Failure with FSLogix 1. Overview When running Electron-based applications in environments using FSLogix Profile or Office Containers , users may encounter issues where the Electron process fails to launch or execute properly . This behavior has been observed in Azure Virtual Desktop (AVD) , Windows Virtual Desktop (WVD) , and other environments where FSLogix filter drivers are active. 2. Symptoms Electron-based applications (e.g., desktop apps built on Electron, CLI wrappers) do not start , remain unresponsive, or terminate silently. No visible logs or error messages are generated by the application. Standard executables run correctly when placed outside of the FSLogix-controlled profile path (e.g., copying to C:\Temp allows execution). The issue is reproducible across all Electron apps in the FSLogix-managed profile. 3. Root Cause The issue is linked to FSLogix filter drivers ( frxdrv , frxdrvvt , frxccd ) interfe...
Read more

KB: Kube-proxy vs CNI Plugin

  kube-proxy is not considered a CNI plugin in Kubernetes or AKS (Azure Kubernetes Service). It serves a different purpose within the Kubernetes networking stack. Let me clarify the roles of kube-proxy and a CNI plugin kube-proxy: Role : kube-proxy manages network rules that allow communication between Kubernetes services and pods. It sets up the networking rules (e.g., iptables, IPVS, or eBPF) to enable service discovery and routing within the cluster. Key Responsibilities : Implements Kubernetes Service networking. Forwards traffic from a service's ClusterIP to the appropriate pod(s) backing the service. Handles load balancing for traffic directed to services. Scope : It operates at the service level, not the pod-to-pod network level. CNI Plugin: Role : A CNI plugin is responsible for setting up the pod network. It ensures that all pods across the cluster can communicate with each other and with the host network. Key Responsibilities : Assigns IP addresses to pods. Configures ...
Read more

KB: Azure ACA Container fails to start (no User Assigned or Delegated Managed Identity found for specified ClientId)

Image
When deploying secure workloads using Azure Container Apps (ACA) , teams often face confusion between User Assigned Managed Identities (UAMI) and App Registrations . While both entities are visible in Azure Active Directory and have similar identifiers (Application ID, Object ID), they serve very different purposes . This confusion can lead to authentication failures when accessing services like Azure App Configuration or Key Vault . A common issue occurs when a container app is configured with a managed identity, but the environment variables or role assignments mistakenly reference an App Registration instead, causing errors like: "No User Assigned or Delegated Managed Identity found for specified ClientId" This article breaks down the difference between UAMI and App Registrations, explains why this issue happens, and outlines the correct approach to resolve it. Symptoms Azure Container App fails to authenticate to Azure App Configuration or other services. Logs...
Read more