Shared Utilities¶

The fr5.shared module contains common utilities and helper functions used across the fr5 package. These tools provide consistent functionality for data handling, visualization, file I/O, and other common tasks.

Overview¶

The shared utilities include:

Data processing functions for cleaning and transforming chemical data
Visualization tools for plotting results and analysis
File I/O utilities for reading/writing various data formats
Validation functions for ensuring data quality
Common constants and configuration settings

Data Processing¶

Data Cleaning¶

Clean and validate chemical datasets:

from fr5.shared import clean_chemical_data

# Clean experimental data
cleaned_data = clean_chemical_data(
    data=raw_experimental_data,
    remove_outliers=True,
    outlier_method='iqr',  # 'iqr', 'zscore', or 'isolation_forest'
    missing_value_strategy='interpolate',  # 'drop', 'interpolate', or 'fill'
    validate_chemistry=True
)

print(f"Cleaned {len(cleaned_data)} rows from {len(raw_experimental_data)}")

Data Transformation¶

Transform data for machine learning:

from fr5.shared import transform_data

# Apply standard transformations
transformed_data = transform_data(
    data=experimental_data,
    transformations={
        'viscosity': 'log',        # Log transform viscosity
        'temperature': 'standardize',  # Z-score standardization
        'pressure': 'normalize'    # Min-max normalization
    }
)

Missing Value Handling¶

Handle missing values intelligently:

from fr5.shared import handle_missing_values

# Multiple strategies for different columns
filled_data = handle_missing_values(
    data=incomplete_data,
    strategies={
        'viscosity': 'interpolate',     # Linear interpolation
        'density': 'knn_impute',       # K-nearest neighbors
        'temperature': 'forward_fill',  # Forward fill
        'catalyst': 'mode'             # Most frequent value
    }
)

Visualization¶

Property Plots¶

Create standardized plots for chemical properties:

from fr5.shared.visualization import plot_property_comparison

# Compare predicted vs experimental values
plot_property_comparison(
    experimental=y_true,
    predicted=y_pred,
    property_name='Viscosity',
    units='mPa·s',
    title='Viscosity Prediction Performance',
    save_path='viscosity_comparison.png'
)

Optimization Progress¶

Visualize Bayesian optimization progress:

from fr5.shared.visualization import plot_optimization_progress

# Plot convergence
plot_optimization_progress(
    optimization_history=bo_history,
    objectives=['yield', 'selectivity'],
    show_pareto_front=True,
    save_path='optimization_progress.png'
)

Molecular Visualization¶

Visualize molecular structures:

from fr5.shared.visualization import plot_molecule

# Plot ionic liquid structure
plot_molecule(
    cation_smiles='CCn1ccnc1',      # EMIM
    anion_smiles='[B-](F)(F)(F)F',  # BF4
    title='EMIM-BF4 Ionic Liquid',
    highlight_interactions=True
)

Descriptor Analysis¶

Visualize descriptor importance and distributions:

from fr5.shared.visualization import plot_descriptor_importance

# Feature importance plot
plot_descriptor_importance(
    importances=feature_importances,
    descriptor_names=descriptor_names,
    top_n=20,
    title='Top 20 Most Important Descriptors'
)

File I/O Utilities¶

Database Operations¶

Common database operations:

from fr5.shared.io import DatabaseManager

# Create database manager
db = DatabaseManager('experiments.db')

# Insert experimental data
db.insert_experiments(
    data=new_experiments,
    table_name='optimization_results'
)

# Query with filters
filtered_data = db.query_experiments(
    table_name='optimization_results',
    filters={'temperature': (250, 350), 'catalyst': 'Pd'},
    order_by='yield DESC',
    limit=100
)

File Format Conversion¶

Convert between different file formats:

from fr5.shared.io import convert_file_format

# Convert CSV to HDF5 for large datasets
convert_file_format(
    input_path='large_dataset.csv',
    output_path='large_dataset.h5',
    input_format='csv',
    output_format='hdf5',
    compression='gzip'
)

# Convert Excel to JSON
convert_file_format(
    input_path='experiments.xlsx',
    output_path='experiments.json',
    input_format='excel',
    output_format='json',
    sheet_name='results'
)

Configuration Management¶

Manage configuration files and settings:

from fr5.shared.config import ConfigManager

# Load configuration
config = ConfigManager('fr5_config.yaml')

# Access settings
model_params = config.get('model_parameters')
data_paths = config.get('data_paths')
optimization_settings = config.get('optimization')

# Update configuration
config.set('optimization.n_iterations', 100)
config.save()

Validation Functions¶

Data Validation¶

Validate chemical data consistency:

from fr5.shared.validation import validate_chemical_data

# Comprehensive data validation
validation_report = validate_chemical_data(
    data=experimental_data,
    required_columns=['cation', 'anion', 'temperature', 'viscosity'],
    value_ranges={
        'temperature': (200, 400),  # Kelvin
        'viscosity': (0.1, 10000)   # mPa·s
    },
    chemistry_validation=True
)

if not validation_report['is_valid']:
    print("Data validation failed:")
    for error in validation_report['errors']:
        print(f"- {error}")

Model Validation¶

Validate machine learning models:

from fr5.shared.validation import validate_model

# Cross-validation with multiple metrics
validation_results = validate_model(
    model=trained_model,
    X=feature_matrix,
    y=target_values,
    cv_folds=5,
    metrics=['rmse', 'mae', 'r2', 'mape'],
    stratify_by='catalyst_type'
)

print(f"CV RMSE: {validation_results['rmse']['mean']:.3f} ± {validation_results['rmse']['std']:.3f}")

Input Validation¶

Validate function inputs:

from fr5.shared.validation import validate_inputs

@validate_inputs
def predict_property(
    cation: str,
    anion: str,
    temperature: float,
    model_type: str = 'catboost'
):
    """
    Predict IL property with input validation.

    Args:
        cation: Cation name (validated against known cations)
        anion: Anion name (validated against known anions)  
        temperature: Temperature in Kelvin (validated range 200-400)
        model_type: Model type (validated against available models)
    """
    # Function implementation...

Common Constants¶

Chemical Constants¶

Access common chemical constants and data:

from fr5.shared.constants import (
    KNOWN_CATIONS, 
    KNOWN_ANIONS,
    PHYSICAL_CONSTANTS,
    UNIT_CONVERSIONS
)

# Check if IL components are supported
if cation in KNOWN_CATIONS and anion in KNOWN_ANIONS:
    print(f"{cation}-{anion} is a supported IL")

# Use physical constants
R = PHYSICAL_CONSTANTS['gas_constant']  # J/(mol·K)
NA = PHYSICAL_CONSTANTS['avogadro']     # mol⁻¹

# Convert units
viscosity_poise = UNIT_CONVERSIONS['viscosity']['mPa_s_to_P'] * viscosity_mpa_s

Model Constants¶

Constants related to models and algorithms:

from fr5.shared.constants import MODEL_DEFAULTS, OPTIMIZATION_DEFAULTS

# Default hyperparameters
catboost_defaults = MODEL_DEFAULTS['catboost']
rf_defaults = MODEL_DEFAULTS['random_forest']

# Default optimization settings  
bo_defaults = OPTIMIZATION_DEFAULTS['bayesian_optimization']

Logging and Debugging¶

Structured Logging¶

Set up consistent logging across the package:

from fr5.shared.logging import setup_logger

# Create logger with standard fr5 format
logger = setup_logger(
    name='my_analysis',
    level='INFO',
    log_file='analysis.log',
    include_console=True
)

logger.info("Starting analysis...")
logger.warning("Temperature outside normal range")
logger.error("Model prediction failed", exc_info=True)

Performance Profiling¶

Profile code performance:

from fr5.shared.profiling import profile_function, profile_block

# Profile a function
@profile_function
def expensive_calculation(data):
    # Complex calculation...
    return results

# Profile a code block
with profile_block("descriptor_generation"):
    descriptors = generate_descriptors(molecules)

# View profiling results
print_profiling_report()

Debug Utilities¶

Debug chemical calculations:

from fr5.shared.debug import debug_prediction, debug_optimization

# Debug a prediction
debug_info = debug_prediction(
    model=trained_model,
    input_data=problematic_input,
    expected_output=known_result,
    tolerance=0.1
)

print(f"Debug analysis: {debug_info}")

# Debug optimization step
debug_optimization(
    bo_manager=optimization_manager,
    iteration=current_iteration,
    save_plots=True
)

Utility Functions¶

Chemical Utilities¶

Common chemical calculations:

from fr5.shared.chemistry import (
    calculate_molecular_weight,
    normalize_smiles,
    validate_smiles,
    fragment_molecule
)

# Calculate molecular weight from SMILES
mw = calculate_molecular_weight('CCn1ccnc1')  # EMIM cation
print(f"Molecular weight: {mw:.2f} g/mol")

# Normalize SMILES representation
canonical_smiles = normalize_smiles('CCN1C=CN=C1')

Statistical Utilities¶

Statistical functions for analysis:

from fr5.shared.stats import (
    confidence_interval,
    bootstrap_statistics,
    correlation_analysis
)

# Calculate confidence interval
ci_lower, ci_upper = confidence_interval(
    predictions,
    confidence_level=0.95,
    method='bootstrap'
)

# Bootstrap statistics
bootstrap_stats = bootstrap_statistics(
    data=experimental_data,
    statistic='mean',
    n_bootstrap=1000,
    confidence_level=0.95
)

Math Utilities¶

Mathematical helper functions:

from fr5.shared.math import (
    safe_log,
    interpolate_missing,
    smooth_data,
    detect_outliers
)

# Safe logarithm (handles zeros and negatives)
log_viscosity = safe_log(viscosity_data, base=10, fill_value=np.nan)

# Smooth noisy data
smoothed_data = smooth_data(
    noisy_measurements,
    method='savgol',  # 'savgol', 'moving_average', 'lowess'
    window_size=5
)

Error Handling¶

Custom Exceptions¶

fr5-specific exception classes:

from fr5.shared.exceptions import (
    ChemistryValidationError,
    ModelPredictionError,
    OptimizationError,
    DataQualityError
)

try:
    prediction = model.predict(input_data)
except ModelPredictionError as e:
    print(f"Prediction failed: {e}")
    # Handle gracefully...

Error Recovery¶

Automatic error recovery for common issues:

from fr5.shared.error_recovery import retry_with_backoff, fallback_model

# Retry failed operations
@retry_with_backoff(max_retries=3, backoff_factor=2.0)
def unreliable_operation():
    # Operation that might fail...
    return result

# Fallback to simpler model on failure
@fallback_model(fallback='linear_regression')
def predict_with_complex_model(X):
    # Try complex model, fall back if it fails
    return predictions

Integration Helpers¶

Pandas Integration¶

Enhanced pandas functionality for chemical data:

from fr5.shared.pandas_extensions import ChemicalDataFrame

# Create enhanced DataFrame
df = ChemicalDataFrame(experimental_data)

# Chemical-aware operations
df.validate_chemistry()
df.normalize_structures()
df.calculate_descriptors(['mw', 'logp'])
df.plot_property('viscosity', color_by='catalyst')

NumPy Integration¶

NumPy utilities for chemical arrays:

from fr5.shared.numpy_extensions import chemical_array

# Create chemical array with metadata
arr = chemical_array(
    descriptor_matrix,
    descriptor_names=feature_names,
    molecule_ids=molecule_names
)

# Enhanced array operations
normalized_arr = arr.normalize_descriptors()
selected_arr = arr.select_features(importance_threshold=0.01)

Best Practices¶

1. Configuration Management¶

Use configuration files for settings rather than hardcoding
Validate configuration parameters on load
Provide sensible defaults for optional parameters

2. Error Handling¶

Use specific exception types for different error categories
Provide helpful error messages with suggested solutions
Implement graceful degradation where possible

3. Logging¶

Use structured logging with consistent formats
Log at appropriate levels (DEBUG, INFO, WARNING, ERROR)
Include context information in log messages

4. Testing¶

Test utility functions with edge cases
Mock external dependencies in tests
Validate outputs match expected formats

5. Performance¶

Cache expensive computations when possible
Use vectorized operations for array processing
Profile code to identify bottlenecks

Example Usage¶

Complete Analysis Pipeline¶

from fr5.shared import *
import pandas as pd

# Set up logging
logger = setup_logger('analysis', level='INFO')

# Load and validate data
logger.info("Loading experimental data...")
data = pd.read_csv('experiments.csv')

validation_report = validate_chemical_data(data)
if not validation_report['is_valid']:
    logger.error("Data validation failed")
    raise DataQualityError(validation_report['errors'])

# Clean data
logger.info("Cleaning data...")
clean_data = clean_chemical_data(
    data,
    remove_outliers=True,
    missing_value_strategy='interpolate'
)

# Transform for modeling
logger.info("Transforming features...")
transformed_data = transform_data(
    clean_data,
    transformations={'viscosity': 'log', 'temperature': 'standardize'}
)

# Visualize results
logger.info("Creating visualizations...")
plot_property_comparison(
    experimental=transformed_data['viscosity_experimental'],
    predicted=transformed_data['viscosity_predicted'],
    property_name='Log Viscosity',
    save_path='results/viscosity_comparison.png'
)

logger.info("Analysis complete!")

This comprehensive shared utilities module provides the foundation for consistent, robust chemical informatics workflows across the fr5 package.