data-science

这段内容是审核时保存的快照。外部来源才是完整且最新的版本。

---
name: data-science
description:
  Data science methodology for EDA, statistical analysis, modeling, and insights generation.
---

# Data Science Skill

Expert methodology for statistical analysis, machine learning, and business insights. Use this skill
when working with data analysis, modeling, or generating insights from datasets.

## Environment Setup

**Python stack (prefer these):**

```python
# Core
import pandas as pd
import numpy as np

# Visualization
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px

# Statistics
from scipy import stats
import statsmodels.api as sm

# ML
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    mean_squared_error, r2_score, confusion_matrix, classification_report
)
```

**For large datasets:**

```python
import polars as pl  # Faster than pandas for large data
import duckdb        # SQL on local files
```

## Exploratory Data Analysis (EDA)

### Quick Data Profile

```python
def quick_profile(df):
    """Generate quick data profile."""
    print(f"Shape: {df.shape}")
    print(f"\nData Types:\n{df.dtypes}")
    print(f"\nMissing Values:\n{df.isnull().sum()}")
    print(f"\nNumeric Summary:\n{df.describe()}")
    print(f"\nCategorical Columns:")
    for col in df.select_dtypes(include=['object', 'category']).columns:
        print(f"  {col}: {df[col].nunique()} unique values")
```

### Distribution Analysis

```python
def analyze_distributions(df, numeric_cols):
    """Check distributions and normality."""
    for col in numeric_cols:
        stat, p_value = stats.normaltest(df[col].dropna())
        skew = df[col].skew()
        kurt = df[col].kurtosis()
        print(f"{col}: skew={skew:.2f}, kurtosis={kurt:.2f}, normal_p={p_value:.4f}")
```

### Correlation Analysis

```python
def correlation_analysis(df, target=None):
    """Analyze correlations, optionally with target."""
    corr_matrix = df.select_dtypes(include=[np.number]).corr()

    if target:
        target_corr = corr_matrix[target].sort_values(ascending=False)
        print(f"Correlations with {target}:\n{target_corr}")

    # Find high correlations (potential multicollinearity)
    high_corr = []
    for i in range(len(corr_matrix.columns)):
        for j in range(i+1, len(corr_matrix.columns)):
            if abs(corr_matrix.iloc[i, j]) > 0.8:
                high_corr.append((corr_matrix.columns[i], corr_matrix.columns[j], corr_matrix.iloc[i, j]))

    if high_corr:
        print(f"\nHigh correlations (>0.8): {high_corr}")

    return corr_matrix
```

### Outlier Detection

```python
def detect_outliers(df, cols, method='iqr'):
    """Detect outliers using IQR or z-score."""
    outliers = {}
    for col in cols:
        if method == 'iqr':
            Q1, Q3 = df[col].quantile([0.25, 0.75])
            IQR = Q3 - Q1
            mask = (df[col] < Q1 - 1.5*IQR) | (df[col] > Q3 + 1.5*IQR)
        else:  # z-score
            z = np.abs(stats.zscore(df[col].dropna()))
            mask = z > 3
        outliers[col] = mask.sum()
    return outliers
```

## Statistical Testing

### Hypothesis Testing Checklist

1. State null and alternative hypotheses
2. Choose significance level (typically α=0.05)
3. Check assumptions (normality, variance homogeneity)
4. Select appropriate test
5. Calculate test statistic and p-value
6. Make decision and interpret

### Common Tests

```python
# t-test (compare two means)
stat, p = stats.ttest_ind(group1, group2)

# Paired t-test (before/after)
stat, p = stats.ttest_rel(before, after)

# ANOVA (compare multiple groups)
stat, p = stats.f_oneway(group1, group2, group3)

# Chi-square (categorical independence)
stat, p, dof, expected = stats.chi2_contingency(contingency_table)

# Mann-Whitney U (non-parametric two groups)
stat, p = stats.mannwhitneyu(group1, group2)

# Correlation significance