data-analysis

The saved excerpt is a snapshot from review. The external source remains the complete and most current version.

---
name: data-analysis
description: Analyze datasets using Python with pandas, numpy, and visualization libraries. Generates statistical summaries, identifies patterns, creates charts, and provides insights. Use when analyzing CSV/Excel files, exploring data, creating visualizations, or when users mention data analysis, statistics, charts, or datasets.
license: MIT
metadata:
  author: agent-skills-demo
  version: "1.0"
  category: data-science
compatibility: Requires Python 3.8+ with pandas, numpy, matplotlib, and seaborn
---

# Data Analysis Skill

## When to Use This Skill

Use this skill when:
- Analyzing datasets (CSV, Excel, JSON)
- Performing statistical analysis
- Creating data visualizations
- Identifying trends and patterns
- Data cleaning and preprocessing
- Users mention "analyze data", "statistics", "charts", "trends", or "insights"

## Analysis Process

### 1. Data Loading & Initial Exploration

**Load the data:**
```python
import pandas as pd
import numpy as np

# CSV files
df = pd.read_csv('data.csv')

# Excel files
df = pd.read_excel('data.xlsx', sheet_name='Sheet1')

# JSON files
df = pd.read_json('data.json')

# From database
import sqlalchemy
engine = sqlalchemy.create_engine('postgresql://user:pass@localhost/db')
df = pd.read_sql('SELECT * FROM table', engine)
```

**Initial exploration:**
```python
# Basic information
print(f"Shape: {df.shape}")
print(f"Columns: {df.columns.tolist()}")
print(f"\nData types:\n{df.dtypes}")
print(f"\nMemory usage: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB")

# First look at data
print("\nFirst 5 rows:")
print(df.head())

# Check for missing values
print("\nMissing values:")
print(df.isnull().sum())

# Basic statistics
print("\nDescriptive statistics:")
print(df.describe())
```

### 2. Data Cleaning

**Handle missing values:**
```python
# Check missing data patterns
missing_pct = (df.isnull().sum() / len(df) * 100).sort_values(ascending=False)
print("Missing data percentage:")
print(missing_pct[missing_pct > 0])

# Drop columns with too many missing values
df = df.drop(columns=missing_pct[missing_pct > 50].index)

# Fill missing values
df['numeric_column'].fillna(df['numeric_column'].median(), inplace=True)
df['categorical_column'].fillna(df['categorical_column'].mode()[0], inplace=True)

# Or drop rows with missing values
df = df.dropna()
```

**Handle duplicates:**
```python
# Check for duplicates
print(f"Duplicate rows: {df.duplicated().sum()}")

# Remove duplicates
df = df.drop_duplicates()

# Keep specific duplicates
df = df.drop_duplicates(subset=['id'], keep='first')
```

**Data type conversions:**
```python
# Convert to datetime
df['date'] = pd.to_datetime(df['date'])

# Convert to numeric
df['price'] = pd.to_numeric(df['price'], errors='coerce')

# Convert to category (saves memory)
df['category'] = df['category'].astype('category')
```

### 3. Statistical Analysis

See [references/STATISTICS.md](references/STATISTICS.md) for detailed formulas.

**Descriptive statistics:**
```python
# Central tendency
mean = df['column'].mean()
median = df['column'].median()
mode = df['column'].mode()[0]

# Dispersion
std = df['column'].std()
variance = df['column'].var()
range_val = df['column'].max() - df['column'].min()
iqr = df['column'].quantile(0.75) - df['column'].quantile(0.25)

# Distribution
skewness = df['column'].skew()
kurtosis = df['column'].kurtosis()

print(f"""
Statistics for {column}:
  Mean: {mean:.2f}
  Median: {median:.2f}
  Std Dev: {std:.2f}
  Range: {range_val:.2f}
  IQR: {iqr:.2f}
  Skewness: {skewness:.2f}
""")
```

**Correlation analysis:**
```python
# Correlation matrix
correlation = df[numeric_columns].corr()
print(correlation)

# Find strong correlations
strong_corr = correlation[(correlation > 0.7) | (correlation < -0.7)]
strong_corr = strong_corr[strong_corr != 1.0].stack()
print("\nStrong correlations:")
print(strong_corr)
```

**Group analysis:**
```python
# Group by categorical variable
grouped = df.groupby('category').agg({
    'sales':