Python Ingest assets now support logging and incremental loads.
You can record messages using the logging module for your Python ingest assets. Access these logs by navigating to the Logs tab in the asset's Build History.
Visualize Python asset logs in the Builg history tab.
Incremental loads
Y42 has introduced support for incremental data loading through the context.state variable in Python-based data ingestion assets. context.state is a dictionary that gets updated with each asset refresh and can store any key-value pair for your data process.
from y42.v1.decorators import data_loader
import requests
import pandas as pd
import json
@data_loader
def todos(context) -> pd.DataFrame:
prev_state = context.state.get()
if prev_state:
last_update = prev_state['last_update']
# perform incremental activities, such as filtering the dataset by last_update variable
else:
# perform full refresh activities
context.state.set({"last_update": datetime.utcnow()})
return df