Agent Skills for Data Engineers

Implements data quality validation with Great Expectations, dbt tests, and data contracts, covering quality dimensions, testing pyramids, and pipeline validation. Directly targets the Quality facet of data engineering.

by wshobson/agents

• Transform
• Quality

Optimizes Apache Spark jobs with partitioning, caching, shuffle tuning, and memory management for large-scale data processing. Core transform and processing skill for data engineers.

by wshobson/agents

Guides end-to-end data warehouse migration to Amazon Redshift, covering discovery, schema/SQL conversion, data migration, validation, and performance comparison. Directly targets the data warehouse task group.

by aws/agent-toolkit-for-aws

• Storage

Covers ClickHouse schema design, MergeTree engines, query optimization, and ingestion for high-performance analytical workloads. Relevant to the Storage and Serving facets of data engineering.

by affaan-m/ecc

• Ingest

Official NVIDIA guidance for cuDF GPU DataFrames, pandas acceleration, dask-cuDF, and ETL operations like joins, groupby, and Parquet I/O. Directly relevant to accelerating data engineering transforms.

by nvidia/skills

• Transform

description: "Azure Storage Services including Blob Storage, File Shares, Queue Storage, Table Storage, and Data Lake. Answers questions about storage access tiers (hot, cool, cold, archive), when to use each tier, and tier comparison. Provides object storage, SMB file shares, async messaging, NoSQL key-value, and big data analytics. Includes lifecycle management. USE FOR:

by microsoft/azure-skills

• Storage

>- Stores, retrieves, and manages data as objects in Cloud Storage (Google Cloud Storage, or GCS) buckets. Use when you need to interact with Cloud Storage — create or configure buckets, upload, download, stream, or transfer data, organize objects with folders, generate signed URLs, control access (IAM, ACLs, public access prevention), set storage classes and tiering (Standard, Nearline, Coldline, Archive), manage cost and lifecycle, protect data (versioning, encryption/CMEK, retention and Bucket Lock, object holds, soft delete), host static websites, trigger Pub/Sub notifications on object changes, mount buckets as a file system (gcsfuse), or optimize storage performance at any scale. Covers the gcloud storage / gsutil CLI, JSON and XML APIs, client libraries, Terraform, and Cloud Storage MCP servers. Don't use for block storage (Persistent Disk), data warehousing/analytics (BigQuery), or databases (Cloud SQL, Spanner, Bigtable, Firestore).

by google/skills

• Ingest
• Transform

Query and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis. WHEN: KQL queries, Kusto database queries, Azure Data Explorer, ADX clusters, log analytics, time series data, IoT telemetry, anomaly detection.

by microsoft/azure-skills

| Azure Data Lake Storage Gen2 SDK for Python. Use for hierarchical file systems, big data analytics, and file/directory operations. Triggers: "data lake", "DataLakeServiceClient", "FileSystemClient", "ADLS Gen2", "hierarchical namespace".

by microsoft/skills

• Storage

>- Creates Cloud Storage (Google Cloud Storage, or GCS) buckets. Analyzes the workload (sensitive data, media hosting, ingestion, web hosting, archiving, backup, logging, analytics, AI/ML, or general-purpose), validates project-level security settings, and designs a secure-by-default, cost-effective configuration (location, storage class, uniform bucket-level access, public access prevention, soft delete, lifecycle) before creating it. Use whenever a user wants to create, make, set up, provision, or spin up a bucket, or needs object storage for an app, service, pipeline, or dataset — even a "simple" or "default" bucket, or when bucket creation is one step in a larger workflow. Outputs or executes the creation via gcloud, the JSON/REST API, Terraform, or SDK client libraries (C++, Java, Python, Go). Don't use for anything other than creating new buckets — for uploads, downloads, access changes, or reconfiguring existing buckets, use google-cloud-storage-basics.

by google/skills

• Ingest
• Storage
• Transform

Create polished, validated architecture, workflow, sequence, data-flow, and lifecycle/state diagrams as explorable standalone HTML with inline SVG, dark/light themes, optional trace motion, and PNG/JPEG/WebP/SVG/WebM export. Accept plain-language requirements or pasted Mermaid flowchart, sequenceDiagram, and stateDiagram input; inspect repository evidence when the diagram must reflect real code. Use when the user asks to visualize system architecture, infrastructure, cloud/security/network topology, technical workflows, API call sequences, request lifecycles, data pipelines, ETL/ELT, data lineage, state machines, or to convert/beautify Mermaid.

by tt-a1i/archify

• Transform

Designs and reviews DynamoDB data layers from access patterns, covering keys, GSIs, Streams, Global Tables, and zero-ETL integrations. Relevant storage-layer design for data engineers on AWS.

by aws/agent-toolkit-for-aws

• Transform

Guides Amazon Timestream for InfluxDB engine selection, provisioning, schema design, and migration for time-series workloads. Relevant storage-layer skill for data engineers handling time-series data.

by aws/agent-toolkit-for-aws

Needs (stated): MCP server

Credit risk data cleaning and variable screening pipeline covering missing rates, IV, PSI, and correlation filtering. Data quality and preprocessing work adjacent to data engineering, but domain-specific to credit modeling.

by github/awesome-copilot

• Transform
• Quality

Builds end-to-end MLOps pipelines from data preparation through training, validation, and deployment with DAG orchestration. Overlaps data engineering on ingestion and orchestration but centers on ML.

by wshobson/agents

• Ingest
• Transform
• Orchestrate

Production ML engineering workflow covering data contracts, reproducible training, evaluation, deployment, and monitoring. Overlaps data engineering on data contracts and pipelines but centers on ML systems.

by affaan-m/ecc

• Transform
• Serving

Establishes a project's quality bar as a written contract and stops agents quietly lowering it. Interviews the user on which dimensions matter, supplies sane default thresholds when they have no number in mind, records everything in CONSTRAINTS.md, and watches the diff for a weakened bar — new @ts-ignore or eslint-disable suppressions, skipped or deleted tests, assertions stripped out, unimplemented stubs, thresholds edited down. Use when no quality bar is written down, when the user says "set up constraints" or "define our standards", when the user wants dimensions they care about — accessibility, web performance, coverage — set up as enforced constraints, when an agent keeps silencing checks or skipping tests to get to green, when you need a coverage or performance threshold and don't know what number to pick, or when an agent writes more code than anyone will read.

by addyosmani/agent-skills

• Transform