Kaggle Home Credit
Этот пример генерирует признаки по кредитной истории из соревнования
Home Credit Default Risk.
Для каждой заявки из application_train.csv в таблице bureau.csv может быть
несколько кредитных записей, поэтому здесь подходит one_to_many_pipeline().
Подготовка данных
Примите правила соревнования, установите и настройте Kaggle API, затем выполните:
kaggle competitions download -c home-credit-default-risk
Expand-Archive home-credit-default-risk.zip -DestinationPath home-credit
Для примера нужны только файлы application_train.csv и bureau.csv.
Генерация признаков
from pathlib import Path
import pandas as pd
from featminer import DerivedColumnsConfig, SelectionConfig, one_to_many_pipeline
data_dir = Path("home-credit")
applications = pd.read_csv(
data_dir / "application_train.csv",
usecols=["SK_ID_CURR", "TARGET"],
)
bureau = (
pd.read_csv(data_dir / "bureau.csv")
.drop(columns="SK_ID_BUREAU")
.merge(applications, on="SK_ID_CURR")
)
features = one_to_many_pipeline(
df=bureau,
aggregation_key="SK_ID_CURR",
target_col="TARGET",
cat_features=["CREDIT_ACTIVE", "CREDIT_CURRENCY", "CREDIT_TYPE"],
derived_columns=DerivedColumnsConfig(max_condition_columns=2),
selection=SelectionConfig.auc(minimum=0.55),
verbose="info",
)
features.to_parquet("home_credit_bureau_features.parquet")
print(features.shape)
SK_ID_CURR связывает кредитные записи с заявкой, а TARGET содержит целевую
переменную. Технический идентификатор записи SK_ID_BUREAU удаляется, чтобы
pipeline не строил по нему признаки. Категориальные колонки перечислены явно:
это позволяет генерировать условные агрегаты, например среднюю сумму кредита
по активным или закрытым договорам.
Результат запуска
Ниже приведён полный вывод запуска.
Input one-to-many frame shape: (1465325, 17)
Target applications with bureau rows: 263491
INFO: [SAMPLING][SKIPPED] cells=24,910,525 <= threshold=100,000,000 (rows=1,465,325, cols=17)
INFO: [ONE_TO_MANY_FORMULAS][CANDIDATES]
Count: 11
Fraction: 11
Max: 144
Mean: 144
Min: 144
Nunique: 36
Std: 144
Sum: 144
INFO: [COMBINATIONS][STAT]
max_condition_columns=2
24 unique aggregation columns
3 unique filtering columns
12 filter column combinations
9 aggregation functions used
778 valid candidates after apply min_category_fraction=0.01
1,465,325 dataset rows
INFO: [ADD ]: 'credit_active__closed__fraction' | AUC=0.5861 | Nulls=0.00%
INFO: [ADD ]: 'credit_active__active__credit_type__consumer_credit__fraction_3b00388f' | AUC=0.5549 | Nulls=0.00%
INFO: [ADD ]: 'credit_active__active__credit_type__credit_card__fraction_6037a9ac' | AUC=0.5505 | Nulls=0.00%
INFO: [ADD ]: 'credit_active__active__count' | AUC=0.5604 | Nulls=0.00%
INFO: [ADD ]: 'days_credit_enddate__credit_type__consumer_credit__max_99cfc7c9' | AUC=0.5611 | Nulls=6.59%
INFO: [ADD ]: 'days_credit_enddate__credit_type__credit_card__max_5a02d512' | AUC=0.5569 | Nulls=42.69%
INFO: [ADD ]: 'amt_credit_sum_debt__credit_type__credit_card__max_339302cd' | AUC=0.5841 | Nulls=38.87%
INFO: [ADD ]: 'days_credit__max' | AUC=0.5803 | Nulls=0.00%
INFO: [ADD ]: 'days_credit__credit_type__credit_card__max_2a6f6fa3' | AUC=0.5920 | Nulls=34.71%
INFO: [ADD ]: 'days_credit_update__credit_type__consumer_credit__max_e0df8a57' | AUC=0.5504 | Nulls=6.32%
INFO: [ADD ]: 'days_credit_update__credit_type__credit_card__max_2d1a15b2' | AUC=0.5777 | Nulls=34.71%
INFO: [DROP]: 'days_credit_enddate__credit_type__credit_card__max_5a02d512'
INFO: [ADD ]: 'days_credit_enddate__sum' | AUC=0.5739 | Nulls=0.00%
INFO: [ADD ]: 'days_credit_enddate__credit_type__consumer_credit__sum_ec43c170' | AUC=0.5858 | Nulls=6.32%
INFO: [DROP]: 'days_credit_enddate__credit_type__consumer_credit__sum_ec43c170'
INFO: [ADD ]: 'days_enddate_fact__sum' | AUC=0.5681 | Nulls=0.00%
INFO: [ADD ]: 'amt_credit_sum_debt__sum' | AUC=0.5514 | Nulls=0.00%
INFO: [DROP]: 'amt_credit_sum_debt__credit_type__credit_card__max_339302cd'
INFO: [ADD ]: 'amt_credit_sum_debt__credit_type__credit_card__sum_00e9a147' | AUC=0.5832 | Nulls=34.71%
INFO: [ADD ]: 'days_credit_update__credit_type__credit_card__sum_c158c4ab' | AUC=0.5525 | Nulls=34.71%
INFO: [ADD ]: 'days_credit_enddate__credit_type__consumer_credit__mean_6b21fa34' | AUC=0.5867 | Nulls=6.59%
INFO: [ADD ]: 'days_enddate_fact__mean' | AUC=0.5639 | Nulls=12.58%
INFO: [ADD ]: 'amt_credit_sum_debt__mean' | AUC=0.5575 | Nulls=2.79%
INFO: [ADD ]: 'amt_credit_sum_debt__credit_type__consumer_credit__mean_24a83e39' | AUC=0.5549 | Nulls=10.14%
INFO: [DROP]: 'days_credit_enddate__credit_type__consumer_credit__mean_6b21fa34'
INFO: [ADD ]: 'days_credit__mean' | AUC=0.6030 | Nulls=0.00%
INFO: [ADD ]: 'days_credit__credit_active__active__mean' | AUC=0.5727 | Nulls=17.59%
INFO: [DROP]: 'days_enddate_fact__mean'
INFO: [ADD ]: 'days_credit__credit_type__consumer_credit__mean_9e4167de' | AUC=0.5892 | Nulls=6.32%
INFO: [DROP]: 'days_credit__credit_type__credit_card__max_2a6f6fa3'
INFO: [ADD ]: 'days_credit__credit_type__credit_card__mean_e03b83de' | AUC=0.5928 | Nulls=34.71%
INFO: [ADD ]: 'days_credit_update__credit_active__closed__mean' | AUC=0.5537 | Nulls=12.65%
INFO: [ADD ]: 'days_credit_enddate__min' | AUC=0.5669 | Nulls=0.85%
INFO: [ADD ]: 'days_credit_enddate__credit_type__credit_card__min_9827bf25' | AUC=0.5512 | Nulls=42.69%
INFO: [ADD ]: 'days_credit_enddate__credit_active__closed__credit_type__consumer_credit__min_f3347751' | AUC=0.5557 | Nulls=16.25%
INFO: [DROP]: 'days_credit_enddate__credit_active__closed__credit_type__consumer_credit__min_f3347751'
INFO: [ADD ]: 'days_enddate_fact__min' | AUC=0.5619 | Nulls=12.58%
INFO: [ADD ]: 'amt_credit_sum__credit_active__active__min' | AUC=0.5526 | Nulls=17.59%
INFO: [ADD ]: 'amt_credit_sum_debt__credit_type__credit_card__min_328ec1af' | AUC=0.5561 | Nulls=38.87%
INFO: [DROP]: 'days_credit_update__credit_type__credit_card__sum_c158c4ab'
INFO: [ADD ]: 'days_credit_update__min' | AUC=0.5692 | Nulls=0.00%
INFO: [DROP]: 'days_credit_update__credit_active__closed__mean'
INFO: [ADD ]: 'days_credit_update__credit_active__closed__min' | AUC=0.5548 | Nulls=12.65%
INFO: [DROP]: 'days_enddate_fact__min'
INFO: [ADD ]: 'days_credit_update__credit_type__consumer_credit__min_d115248d' | AUC=0.5677 | Nulls=6.32%
INFO: [ADD ]: 'days_credit__std' | AUC=0.5578 | Nulls=13.69%
INFO: 778 valid combinations were checked
68 features were dropped due to high correlation
Generated shape: (263491, 27)
Saved features: home_credit_bureau_features.parquet
Pipeline оставил 26 сгенерированных признаков и колонку TARGET. Лучший
одиночный признак — days_credit__mean с AUC 0.6030.