KB: Airflow Operators

In Apache Airflow, operators are tasks that perform specific actions in a workflow or Directed Acyclic Graph (DAG). Each operator defines a particular action or job, such as running a script, transferring files, or interacting with external systems. Airflow provides various types of operators, which can be categorized broadly as follows:


1. Basic Operators

  • PythonOperator: Executes Python functions directly.
  • BashOperator: Runs Bash commands.
  • DummyOperator: Used as a placeholder or for creating dependencies.

2. Transfer Operators

  • Used to transfer data between different systems or databases.
  • Examples: S3ToRedshiftOperator (transfers data from S3 to Redshift), MySqlToS3Operator, S3ToSnowflakeOperator.

3. Sensors

  • Special operators that wait for a specific condition to be true.
  • Examples: FileSensor (waits for a file to be present), ExternalTaskSensor (waits for another task to complete), TimeDeltaSensor (waits for a specified time).

4. Hooks

  • Not exactly operators, but hooks allow operators to connect to external systems (e.g., databases, APIs).
  • Commonly used with custom operators to extend functionality.

5. Custom Operators

  • You can create custom operators by subclassing BaseOperator, which is helpful when standard operators don’t meet specific requirements.

6. ETL Operators

  • Specific operators for ETL processes, like BigQueryOperator or HiveOperator.

7. Utility Operators

  • Used for more advanced workflow control.
  • BranchPythonOperator: Chooses which path to take in the DAG.
  • SubDagOperator: Allows running a DAG as part of another DAG.

Each operator has its own set of parameters, allowing customization, and can be chained together to define complex workflows. These are essential for building workflows that manage data pipelines and automate tasks effectively.


 Airflow Operators: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html


Comments

Post a Comment

Popular posts from this blog

KB: Azure ACA Container fails to start (no User Assigned or Delegated Managed Identity found for specified ClientId)

Image
When deploying secure workloads using Azure Container Apps (ACA) , teams often face confusion between User Assigned Managed Identities (UAMI) and App Registrations . While both entities are visible in Azure Active Directory and have similar identifiers (Application ID, Object ID), they serve very different purposes . This confusion can lead to authentication failures when accessing services like Azure App Configuration or Key Vault . A common issue occurs when a container app is configured with a managed identity, but the environment variables or role assignments mistakenly reference an App Registration instead, causing errors like: "No User Assigned or Delegated Managed Identity found for specified ClientId" This article breaks down the difference between UAMI and App Registrations, explains why this issue happens, and outlines the correct approach to resolve it. Symptoms Azure Container App fails to authenticate to Azure App Configuration or other services. Logs...
Read more

Electron Process Execution Failure with FSLogix

  Technical Note: Electron Process Execution Failure with FSLogix 1. Overview When running Electron-based applications in environments using FSLogix Profile or Office Containers , users may encounter issues where the Electron process fails to launch or execute properly . This behavior has been observed in Azure Virtual Desktop (AVD) , Windows Virtual Desktop (WVD) , and other environments where FSLogix filter drivers are active. 2. Symptoms Electron-based applications (e.g., desktop apps built on Electron, CLI wrappers) do not start , remain unresponsive, or terminate silently. No visible logs or error messages are generated by the application. Standard executables run correctly when placed outside of the FSLogix-controlled profile path (e.g., copying to C:\Temp allows execution). The issue is reproducible across all Electron apps in the FSLogix-managed profile. 3. Root Cause The issue is linked to FSLogix filter drivers ( frxdrv , frxdrvvt , frxccd ) interfe...
Read more

KB:RMM VS DEX (Remote Monitoring Management vs Digital Employee Experience)

Digital Employee Experience (DEX) and Remote Monitoring and Management (RMM) tools serve different purposes and cater to distinct aspects of IT and employee management. Digital Employee Experience (DEX) Focus: Enhancing the overall experience of employees with their digital workplace tools and environment. Key Features: User Experience Monitoring: Tracks how employees interact with software and hardware, identifying issues impacting productivity. Performance Analytics: Provides insights into the performance of applications from the end-user's perspective. Feedback Mechanisms: Allows employees to report issues and give feedback on their digital tools and environment. Employee Well-being: Monitors factors that affect employee satisfaction and well-being, such as system performance and usability. Proactive Support: Identifies potential issues before they become significant problems, enabling proactive IT support. Goals: Improve employee satisfaction and productivity. Ensure a sm...
Read more