data-engineering-agent-skills

In the author's own words, from each skill's SKILL.md — showing the first 30 of 74 files.

• airflow-and-workflow-orchestration

  Guides agents through workflow orchestration design and operation across Airflow-style DAGs, cloud-native schedulers, and event-driven pipeline control planes. Use when building or modifying workflow dependencies, retries, triggers, sensors, SLAs, or cross-system pipeline coordination.

  skills/airflow-and-workflow-orchestration/SKILL.md on GitHub (opens in a new tab)

• apache-beam-unified-batch-and-stream

  Guides agents through Apache Beam pipelines that unify batch and streaming logic. Use when designing Beam transforms, windowing, runners, replay behavior, or portability across execution backends.

  skills/apache-beam-unified-batch-and-stream/SKILL.md on GitHub (opens in a new tab)

• apache-hudi-lakehouse

  Guides agents through Apache Hudi lakehouse design. Use when managing incremental upserts, record-level mutations, timeline behavior, compaction, and Hudi-based lakehouse tables.

  skills/apache-hudi-lakehouse/SKILL.md on GitHub (opens in a new tab)

• api-and-saas-ingestion-patterns

  Guides agents through API and SaaS ingestion workflows. Use when extracting data from REST, GraphQL, or SaaS platforms with pagination, rate limits, auth rotation, backfills, or unstable source contracts.

  skills/api-and-saas-ingestion-patterns/SKILL.md on GitHub (opens in a new tab)

• avro-protobuf-json-schema-registry

  Guides agents through schema-registry-backed event contracts. Use when managing Avro, Protobuf, or JSON Schema for event streams, compatibility policies, producer and consumer evolution, or contract enforcement in messaging systems.

  skills/avro-protobuf-json-schema-registry/SKILL.md on GitHub (opens in a new tab)

• bigquery-and-dataform-platform-engineering

  Guides agents through BigQuery- and Dataform-centered data engineering workflows. Use when designing BigQuery physical models, ingestion boundaries, Dataform transformation workflows, slot or cost controls, and platform decisions across BigQuery, Dataflow, Dataproc, and GCP orchestration services.

  skills/bigquery-and-dataform-platform-engineering/SKILL.md on GitHub (opens in a new tab)

• cdc-and-incremental-loading

  Guides agents through change data capture and incremental load design. Use when building or modifying watermark-based loads, upserts, deduplication, merge logic, late data handling, or replayable incremental pipelines.

  skills/cdc-and-incremental-loading/SKILL.md on GitHub (opens in a new tab)

• clickhouse-real-time-analytics

  Guides agents through ClickHouse-based real-time analytics design. Use when building fast analytical serving layers, event aggregations, materialized views, or low-latency metric access patterns.

  skills/clickhouse-real-time-analytics/SKILL.md on GitHub (opens in a new tab)

• data-catalog-and-discovery

  Guides agents through data catalog, discovery, and metadata quality workflows. Use when publishing datasets, improving discoverability, curating lineage metadata, or making data products easier for other teams to find and trust.

  skills/data-catalog-and-discovery/SKILL.md on GitHub (opens in a new tab)

• data-contract-testing-with-schema-registry

  Guides agents through data-contract testing using schema registries and compatibility checks. Use when validating event contracts, stream schema evolution, consumer compatibility, or release gates for schema-managed systems.

  skills/data-contract-testing-with-schema-registry/SKILL.md on GitHub (opens in a new tab)

• data-lake-and-zone-architecture

  Guides agents through data lake and zone architecture design. Use when defining raw, refined, curated, or publish layers; storage organization; retention; and operational boundaries for a data lake.

  skills/data-lake-and-zone-architecture/SKILL.md on GitHub (opens in a new tab)

• data-mesh-and-domain-oriented-design

  Guides agents through domain-oriented data product and data mesh design. Use when organizing ownership, domain boundaries, federated governance, and shared platform responsibilities across multiple teams.

  skills/data-mesh-and-domain-oriented-design/SKILL.md on GitHub (opens in a new tab)

• data-migration-and-platform-cutover

  Guides agents through data migration and platform cutover workflows. Use when moving pipelines, tables, contracts, orchestration, or workloads between systems, clouds, warehouses, lakehouses, or serving layers.

  skills/data-migration-and-platform-cutover/SKILL.md on GitHub (opens in a new tab)

• data-observability-and-sla-management

  Guides agents through data observability and service-level management. Use when defining or improving freshness, completeness, anomaly detection, alerting, lag tracking, run metadata, and ownership for production data products.

  skills/data-observability-and-sla-management/SKILL.md on GitHub (opens in a new tab)

• data-platform-ci-cd-and-release-management

  Guides agents through CI/CD and release management for data platforms. Use when promoting pipeline code, SQL models, contracts, infra, or configuration across environments with validation gates, staged rollout, and rollback awareness.

  skills/data-platform-ci-cd-and-release-management/SKILL.md on GitHub (opens in a new tab)

• data-platform-disaster-recovery-and-business-continuity

  Guides agents through disaster recovery and business continuity planning for data platforms. Use when defining region or account failover, backup and restore, RTO or RPO targets, control-plane recovery, or restore drills for pipelines, warehouses, lakehouses, and publish surfaces.

  skills/data-platform-disaster-recovery-and-business-continuity/SKILL.md on GitHub (opens in a new tab)

• data-platform-operating-model-and-service-ownership

  Guides agents through data platform operating model and ownership design. Use when defining platform team responsibilities, service tiers, golden paths, escalation boundaries, onboarding flows, or handoffs between central platform teams and domain or product teams.

  skills/data-platform-operating-model-and-service-ownership/SKILL.md on GitHub (opens in a new tab)

• data-quality-and-contract-testing

  Drives data implementation with contracts, assertions, and validation evidence. Use when adding or changing ingestion logic, transformations, schemas, or published data products.

  skills/data-quality-and-contract-testing/SKILL.md on GitHub (opens in a new tab)

• data-quality-platforms-and-rule-management

  Guides agents through data-quality operating models and tool selection. Use when designing rule portfolios, severity levels, ownership, evidence, and enforcement across dbt tests, Great Expectations, Deequ, Cuallee, Soda, warehouse-native checks, and platform monitoring workflows.

  skills/data-quality-platforms-and-rule-management/SKILL.md on GitHub (opens in a new tab)

• data-reconciliation-and-financial-controls

  Guides agents through reconciliation and control design for business-critical data. Use when validating financial, operational, or audit-sensitive metrics with source-to-target totals, control balances, exception tracking, or close-process dependencies.

  skills/data-reconciliation-and-financial-controls/SKILL.md on GitHub (opens in a new tab)

• data-resiliency-testing-and-failure-injection

  Guides agents through resiliency testing for data platforms. Use when designing or running failure drills, recovery validation, failover tests, replay-safety checks, dependency outage exercises, or fault injection for pipelines and publishes.

  skills/data-resiliency-testing-and-failure-injection/SKILL.md on GitHub (opens in a new tab)

• data-security-compliance-and-regulated-data

  Guides agents through regulated-data security and compliance workflows for PII, PCI, HIPAA, PHI, and similar obligations. Use when data products handle sensitive fields, regulated records, control evidence, or audit-bound publish paths.

  skills/data-security-compliance-and-regulated-data/SKILL.md on GitHub (opens in a new tab)

• data-sharing-and-publishing-contracts

  Guides agents through publishing data products for internal or external consumers. Use when sharing tables, files, extracts, APIs, or reverse-ETL-ready outputs that require stable contracts, ownership, and communication.

  skills/data-sharing-and-publishing-contracts/SKILL.md on GitHub (opens in a new tab)

• data-specification

  Creates structured specifications for data products and pipeline changes. Use when starting a new pipeline, model, ingestion flow, or any significant change with unclear requirements.

  skills/data-specification/SKILL.md on GitHub (opens in a new tab)

• dataplex-and-bigquery-governance

  Guides agents through GCP-native data governance workflows with Dataplex and BigQuery. Use when designing lakes, zones, policy tags, metadata quality, lineage, discovery, and governed publishing across Cloud Storage, BigQuery, Dataflow, Dataproc, and Google Cloud analytics platforms.

  skills/dataplex-and-bigquery-governance/SKILL.md on GitHub (opens in a new tab)

• dbt-and-analytics-engineering

  Guides agents through analytics engineering workflows with dbt. Use when building or modifying staging models, marts, tests, snapshots, documentation, exposures, or semantic-layer-facing models.

  skills/dbt-and-analytics-engineering/SKILL.md on GitHub (opens in a new tab)

• debezium-and-kafka-connect-cdc

  Guides agents through Debezium and Kafka Connect CDC workflows. Use when streaming database changes into Kafka topics, managing connectors, snapshots, schema evolution, or downstream CDC consumers.

  skills/debezium-and-kafka-connect-cdc/SKILL.md on GitHub (opens in a new tab)

• delta-lake-and-medallion-architecture

  Guides agents through Delta Lake and medallion-style lakehouse design. Use when building or modifying bronze, silver, and gold layers, Delta Lake mutation patterns, streaming-to-batch lakehouse flows, or Databricks-centered serving architectures.

  skills/delta-lake-and-medallion-architecture/SKILL.md on GitHub (opens in a new tab)

• duckdb-local-analytics-and-dev

  Guides agents through DuckDB-based local analytics and development workflows. Use when prototyping models locally, validating transformations, reproducing data issues quickly, or building lightweight analytical tooling without a full warehouse.

  skills/duckdb-local-analytics-and-dev/SKILL.md on GitHub (opens in a new tab)

• enterprise-etl-and-data-integration-modernization

  Guides agents through operating, hardening, and modernizing enterprise ETL and integration stacks such as Informatica, Talend, DataStage, SSIS, and Matillion. Use when legacy mappings, job orchestration, migration, or coexistence with modern lakehouse patterns must be handled safely.

  skills/enterprise-etl-and-data-integration-modernization/SKILL.md on GitHub (opens in a new tab)

No skill matches that filter.

Similar skills

• Wazaby tw93 🥷 Engineering habits you already know, turned into skills AI assistants like Claude Code, Codex and Cursor can run. 7.3k stars · 8 skills
• ponytailby DietrichGebert Makes your AI agent think like the laziest senior dev in the room. The best code is the code you never wrote. 164k stars · 6 skills
• taste-skillby Leonxlnx Agent skills that give coding agents taste: art direction, reference images and a browser check instead of generic AI pages 97.3k stars · 13 skills
• awesome-claude-skillsby ComposioHQ A curated list of awesome Claude Skills, resources, and tools for customizing Claude AI workflows 77.2k stars
• marketingskillsby coreyhaines31 Marketing skills for Claude Code and AI agents. CRO, copywriting, SEO, analytics, and growth engineering. 54.9k stars · 30 skills
• agentsby wshobson Multi-harness agentic plugin marketplace for Claude Code, Codex, Cursor, OpenCode, GitHub Copilot, Google Antigravity, and Pi 40.9k stars · 30 skills