data-engineering

これはレビュー時に保存された抜粋です。完全で最新の内容は外部ソースを確認してください。

---
# ═══════════════════════════════════════════════════════════════════════════
# SKILL: Data Engineering
# Version: 2.0.0 | Updated: 2025-01
# ═══════════════════════════════════════════════════════════════════════════
name: data-engineering
description: Data engineering, machine learning, AI, and MLOps. From data pipelines to production ML systems and LLM applications.

# ACTIVATION TRIGGERS
triggers:
  - data engineering
  - machine learning
  - ml
  - ai
  - mlops
  - spark
  - airflow
  - llm
  - rag
  - langchain

# SKILL PARAMETERS
parameters:
  role:
    type: string
    enum: [data-engineer, ml-engineer, ai-engineer]
    required: true
  experience:
    type: string
    enum: [beginner, intermediate, advanced]
    required: false
    default: beginner

# OUTPUT SPECIFICATION
outputs:
  learning_path:
    type: array
  tech_stack:
    type: object
  projects:
    type: array

# RELIABILITY
retry:
  max_attempts: 3
  backoff: exponential

# OBSERVABILITY
observability:
  log_level: info
  metrics: [path_completion_rate]

level: advanced
prerequisites:
  - programming-basics
  - python-advanced

sasmp_version: "1.3.0"
bonded_agent: 01-core-paths
bond_type: PRIMARY_BOND
---

# Data Engineering Skill

## Quick Reference

| Role | Focus | Timeline | Entry From |
|------|-------|----------|------------|
| **Data Engineer** | Pipelines, Infra | 12-24 mo | Backend Dev |
| **ML Engineer** | Models, Features | 12-24 mo | Data Scientist |
| **AI Engineer** | LLMs, Agents | 6-12 mo | Any Developer |

---

## Learning Paths

### Data Engineer
```
[1] SQL Mastery (4-6 wk)
 │  └─ Window functions, CTEs, optimization
 │
 ▼
[2] Python for Data (4-6 wk)
 │  └─ Pandas, file formats, scripting
 │
 ▼
[3] ETL/ELT Pipelines (6-8 wk)
 │  └─ Extract, transform, load patterns
 │
 ▼
[4] Big Data: Spark (8-12 wk)
 │  └─ PySpark, DataFrames, partitioning
 │
 ▼
[5] Data Warehouse (4-6 wk)
 │  └─ Star schema, dbt, Snowflake/BQ
 │
 ▼
[6] Orchestration (4-6 wk)
    └─ Airflow/Prefect, scheduling, monitoring
```

**2025 Stack:** Python + Spark + Airflow + dbt + Snowflake/BigQuery

---

### ML Engineer
```
[1] Python + NumPy (4-6 wk)
 │
 ▼
[2] Math Foundations (6-8 wk)
 │  └─ Linear algebra, calculus, statistics
 │
 ▼
[3] Classical ML (8-12 wk)
 │  └─ scikit-learn, XGBoost, evaluation
 │
 ▼
[4] Deep Learning (8-12 wk)
 │  └─ PyTorch, CNNs, Transformers
 │
 ▼
[5] MLOps (6-8 wk)
    └─ MLflow, model serving, monitoring
```

**2025 Stack:** Python + PyTorch + scikit-learn + MLflow + W&B

---

### AI Engineer (2025 Hot Path)
```
[1] LLM Fundamentals (2-3 wk)
 │  └─ Tokens, embeddings, context windows
 │
 ▼
[2] Prompt Engineering (2-3 wk)
 │  └─ Few-shot, CoT, structured output
 │
 ▼
[3] RAG Systems (3-4 wk)
 │  └─ Embeddings, vector DBs, retrieval
 │
 ▼
[4] AI Agents (4-6 wk)
 │  └─ Tool calling, agent loops, memory
 │
 ▼
[5] Production Deploy (ongoing)
    └─ Evaluation, guardrails, monitoring
```

**2025 Stack:** Python + LangChain/LlamaIndex + OpenAI/Anthropic + ChromaDB

---

## 2025 Tool Matrix

### Data Processing
| Tool | Scale | Use Case |
|------|-------|----------|
| **Pandas** | <10GB | Prototyping, small data |
| **Polars** | <100GB | Fast local processing |
| **Spark** | >100GB | Distributed processing |
| **dbt** | Any | Transformations, testing |

### ML Frameworks
| Framework | Best For | Complexity |
|-----------|----------|------------|
| **scikit-learn** | Classical ML | Low |
| **XGBoost** | Tabular data | Low |
| **PyTorch** | Research, flexibility | Medium |
| **TensorFlow** | Production, mobile | Medium |

### LLM/AI Tools
| Tool | Use Case |
|------|----------|
| **LangChain** | LLM orchestration |
| **LlamaIndex** | RAG systems |
| **Claude/OpenAI** | LLM APIs |
| **ChromaDB** | Vector storage |

---

## Algorithm Reference

### Classical ML
| Type | Algorithms |
|------|------------|
| Regression | Linear, Ridge, Lasso, ElasticNet |
| Classification | Logistic, SVM, Decision Tree |
| Ensemble | Random Forest, XGBoost, LightGBM |