Feature Engineering in the Age of AI

For decades, feature engineering meant manually crafting numeric variables from raw data. An experienced data scientist would spend weeks creating features like price_per_square_foot, days_since_last_purchase, or interaction_term_age_income.

Then embeddings arrived. A single call to a pre-trained model could generate 768 dimensions of semantic meaning from a text snippet.

Then LLMs arrived. A prompt could extract structured features from unstructured documents in seconds.

Which approach should you use? The answer depends on your data, your constraints, and your goals.

Classical Feature Engineering

Classical feature engineering transforms raw data into numeric features using domain knowledge and mathematical formulas.

Examples

import pandas as pd
import numpy as np

# Raw data
df = pd.DataFrame({
    'price': [200000, 350000, 180000],
    'sqft': [1200, 1800, 950],
    'age_years': [25, 10, 40],
    'bedrooms': [3, 4, 2],
    'neighborhood': ['A', 'B', 'A']
})

# Classical feature engineering
df['price_per_sqft'] = df['price'] / df['sqft']
df['bedrooms_per_sqft'] = df['bedrooms'] / df['sqft']
df['age_squared'] = df['age_years'] ** 2  # Non-linear
df['is_new'] = (df['age_years'] < 5).astype(int)
df['price_x_bedrooms'] = df['price'] * df['bedrooms']  # Interaction

Strengths

Limitations

• Requires deep domain expertise
• Cannot capture complex patterns automatically
• Time-consuming to create and maintain
• Limited to patterns you can think of

Embedding-Based Features

Embeddings are dense vector representations learned by neural networks. They capture semantic meaning that classical features miss.

Text Embeddings

from sentence_transformers import SentenceTransformer

# Load a pre-trained embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Raw text data
texts = [
    "The product works great and arrived on time",
    "Terrible quality, broke after one day",
    "Average product, nothing special"
]

# Generate embeddings (384 dimensions)
embeddings = model.encode(texts)
print(f"Shape: {embeddings.shape}")  # (3, 384)

# Use as features in ML model
from sklearn.ensemble import RandomForestClassifier

# embeddings now contains 384 features per text
# X = embeddings, y = sentiment labels

Image Embeddings

import torch
from torchvision import models, transforms
from PIL import Image

# Load pre-trained ResNet (remove classification head)
resnet = models.resnet50(pretrained=True)
feature_extractor = torch.nn.Sequential(*list(resnet.children())[:-1])

# Preprocessing
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                       std=[0.229, 0.224, 0.225])
])

# Extract features (2048 dimensions)
img = Image.open('photo.jpg')
img_tensor = transform(img).unsqueeze(0)
features = feature_extractor(img_tensor)
print(f"Shape: {features.shape}")  # (1, 2048, 1, 1)

Strengths

Limitations

• High dimensionality (128-3072 features)
• Hard to interpret individual dimensions
• Requires GPU for large models
• May not capture domain-specific patterns

LLM-Derived Features

LLMs can generate structured features from unstructured data using prompts. This is the newest approach.

Sentiment and Emotion Extraction

import openai
import json

def extract_features_with_llm(text: str) -> dict:
    """Use LLM to extract structured features from text."""
    
    prompt = f"""Extract the following features from this review:
    
Text: "{text}"

Return JSON with these fields:
- sentiment: positive/negative/neutral
- emotion: joy/anger/sadness/surprise/disgust
- urgency: high/medium/low
- topic: product/service/delivery/price
- key_phrases: list of 2-3 key phrases
"""
    
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    
    return json.loads(response.choices[0].message.content)

# Example
features = extract_features_with_llm(
    "My order arrived damaged and customer support was unhelpful"
)
# {'sentiment': 'negative', 'emotion': 'anger', 
#  'urgency': 'high', 'topic': 'delivery',
#  'key_phrases': ['arrived damaged', 'unhelpful support']}

Batch Feature Extraction

import pandas as pd
from tqdm import tqdm

def batch_extract_features(texts: list, batch_size=10) -> pd.DataFrame:
    """Extract features for multiple texts efficiently."""
    
    results = []
    for i in tqdm(range(0, len(texts), batch_size)):
        batch = texts[i:i+batch_size]
        for text in batch:
            features = extract_features_with_llm(text)
            results.append(features)
    
    return pd.DataFrame(results)

# Extract features for 1000 reviews
df_features = batch_extract_features(df['review_text'].tolist())

Strengths

Limitations

• API cost per sample ($0.01-0.10+ per call)
• Latency at inference (100ms-2s per sample)
• Non-deterministic outputs
• Rate limits and quotas

When to Use Each Approach

The Hybrid Approach

The most effective modern pipelines combine all three:

import pandas as pd
import numpy as np
from sentence_transformers import SentenceTransformer

# Classical features
df['text_length'] = df['text'].str.len()
df['word_count'] = df['text'].str.split().str.len()
df['avg_word_length'] = df['text'].apply(
    lambda x: np.mean([len(w) for w in x.split()])
)

# Embedding features
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(df['text'].tolist())
embedding_df = pd.DataFrame(
    embeddings, 
    columns=[f'emb_{i}' for i in range(embeddings.shape[1])]
)

# Combine all features
X = pd.concat([df[['text_length', 'word_count', 'avg_word_length']], 
                embedding_df], axis=1)

print(f"Final feature count: {X.shape[1]}")
# 387 features (3 classical + 384 embeddings)

Feature Store and Caching

For production systems, cache computed features:

import hashlib
import pickle
from pathlib import Path

class FeatureCache:
    def __init__(self, cache_dir='features_cache'):
        self.cache_dir = Path(cache_dir)
        self.cache_dir.mkdir(exist_ok=True)
    
    def get_key(self, text: str) -> str:
        return hashlib.md5(text.encode()).hexdigest()
    
    def get_features(self, text: str):
        key = self.get_key(text)
        cache_file = self.cache_dir / f'{key}.pkl'
        
        if cache_file.exists():
            with open(cache_file, 'rb') as f:
                return pickle.load(f)
        
        # Compute and cache
        embedding = model.encode([text])[0]
        with open(cache_file, 'wb') as f:
            pickle.dump(embedding, f)
        
        return embedding

Try It Yourself

• Data Leakage in ML — Avoid feature leakage in your pipeline
• Train/Val/Test Sets — Split correctly before engineering features
• Python Optimization — Handle large feature sets efficiently

Further Reading

• Analytics Vidhya: Feature Engineering with LLMs
• ML Mastery: LLM Embeddings for Time Series
• Scikit-learn: Preprocessing
• BestWordz: Data Leakage in ML

Conclusion

Feature engineering has evolved from a purely manual process to a multi-tool approach. Classical features remain essential for interpretable, fast, and cost-effective models. Embeddings capture semantic meaning that no human could manually encode. LLM-derived features provide flexible, zero-shot extraction from unstructured data.

The key insight: these approaches are not mutually exclusive. The best modern pipelines combine classical domain knowledge with embedding power and LLM flexibility.

Start with classical features when you have domain expertise. Add embeddings for semantic understanding. Use LLM-derived features for prototyping and complex extraction tasks. Cache everything in production.

Feature engineering is no longer just about creating features. It is about choosing the right method for the right data.