Model Framework Core Components
This document describes the key components and concepts of the Credmark Model Framework.
A Credmark model inherits from a simple base class called
Model. The actual code can be found here.
All Models should import this class
from credmark.cmf.model import Model, subclass it, and override the run() method. See examples here.
@Model.describe() provides a simple interface to define the model properties such as slug, version, display_name, description, category, developer, input, output etc so that it can be used easily by consumers and other models.
If description is not specified, the
__doc__ string of the model’s class is used for the model description.
See example here.
Data Transfer Object (DTO)
Input and output data for models are json-serializable objects of arbitrary depth and complexity. Objects can have 0 or more keys whose values can be null, number, string, array, or another object.
Although you can use dictionaries for model input and output data in your python code, we strongly encourage the use of DTOs (Data Transfer Objects.)
DTOs are classes with typed properties which will serialize and deserialize to and from JSON. They also automatically produce a JSON-schema that is used to document the input and output of a model. Each model may have their own DTOs or may share or inherit a DTO from another model that you have developed.
To create a DTO, simply subclass the DTO base class and use DTOFields to annotate your properties. Under the hood, the Credmark Model Framework uses the pydantic python module (DTO is simply an alias for pydantic BaseModel and DTOField an alias for Field) so almost anything that works with pydantic will work for your DTO.
Please see the pydantic docs for more information.
Model Error Detail DTO [Advanced Topic]
Besides input and output, subclasses of
ModelBaseError can use a DTO for the
data.detail object instead of a dict. You can simply pass a DTO as the
detail arg in a model constructor:
address = Address(some_address_string) e = ModelDataError(message='Address is not a contract', code=ModelDataError.Codes.CONFLICT, detail=address)
If your detail object has many properties and you want to document the error and details, you can create a custom DTO and error class:
Create a DTO subclass that defines the data you want to store in the detail.
class TokenAddressNotFoundDetailDTO(DTO): address: Address = DTOField(...,description='Address for token not found')
Create a DTO subclass that defines the new error DTO. (This step is not strictly necessary but it lets you document the error.) The trick is to use the generic properties of the
ModelErrorDTOto specify the detail’s DTO class:
class TokenAddressNotFoundDTO(ModelErrorDTO[TokenAddressNotFoundDetailDTO]): """ This error occurs when there is no token at the specified address. The detail contains the address. """
Then create a
ModelRunError) subclass and set the class property
dto_classto your new error DTO class:
class TokenAddressNotFoundError(ModelDataError): dto_class = TokenAddressNotFoundDTO
You can now create an error instance with:
# bad_address is set to an Address instance error = TokenAddressNotFoundError(message='Bad address', detail=TokenAddressNotFoundDetailDTO(address=bad_address)) # You can now access: error.data.detail.address
We have some built-in reusable data type classes available under Credmark.cmf.types.
We have created and grouped together different classes to manage input and output data types (DTOs) to be used by models. These types include some standard blockchain and financial data structures as well as some standard input and output objects for Credmark models.
1. Address: this class is a subclass of string and holds a blockchain address.
Address class is inherited from
str to help with web3 address conversion. It’s highly recommended to use it instead of a string.
✔️: Address(“0x7d2768dE32b0b80b7a3454c06BdAc94A69DDc7A9”).checksum # checksum version to be used
❌: Address(“0x7d2768dE32b0b80b7a3454c06BdAc94A69DDc7A9”) # lower case version
❌:”0x7d2768de32b0b80b7a3454c06bdac94a69ddc7a9” # lower case version
from credmark.cmf.types import Address, Contract contract = Contract( # lending pool address address=Address("0x7d2768dE32b0b80b7a3454c06BdAc94A69DDc7A9").checksum, abi=AAVE_V2_TOKEN_CONTRACT_ABI )
The address can be provided in lower case, upper case or checksum hex format. This class will normalize the address into lower case. Note that It can be used as a normal string but it also has a “checksum” property which returns a web3 ChecksumAddress.
See e_03_address.py on how to use this class.
Account simply holds an address. Accounts is a list of account instances which allows iteration through each account.
See e_04_account.py on how to use this class.
3. Contract: a
Contract is a subclass of
Account which has a name, deployed transaction hash, abi, protocol name etc.
Object instantiation of this class will load all information available for the contract (against contract address provided as input) in our database and you can access whatever information you want from the object.
Token is a specific kind of contract; hence the Token class inherits from
This class allows you to load token information with an address or symbol as well as get its price in USD Currently this class supports data load for erc20 token but we will support erc721 as well soon.
See e_06_token.py on how to use this class. Token_data.py lists all erc20 tokens currently supported.
5. Price: The
Price classes can be used to hold a price.
6. Position: A
Position class holds a
Token and an amount. It can calculate its value based on the token price in USD. You can also access the scaled amount property if you need the scaled amount of the erc20 token.
Token_data.py lists all erc20 tokens currently supported.
7. Portfolio: A
Portfolio class holds a list of
Position instances. So, it can be used to calculate all positions within a wallet.
Each model runs with a particular context, including the block chain id, block number, and a configured web3 instance (among other things). The context’s web3 instance can be used to make RPC calls. It also enforces deterministic behavior for Models.
ModelContext class is the context for the model and can be accessed from a model as
The base code can be found here. It provides an interface for models to run other models, call contracts, get ledger data, use a web3 instance etc.
The key utilities in
Calling Other Models
A model can call other models and use their results. You can pass the input as an input arg and the model output is returned as a dict (or DTO if
return_type is specified.)
If an error occurs during a call to run a model, an exception is raised. See Error handling
There are 2 ways to call another model:
Models are exposed on
context.models by their slug (with any “-” (hyphens) in the slug replaced with “_” (underscores)) and can be called like a function, passing the input as a DTO or dict or as standard keyword args (kwargs).
For example, here we use keyword args:
# Returns a dict with output of the model result = self.context.models.example.model(message='Hello world')
You can use a DTO for the output by initializing it with the output dict.
Here we use a DTO instance as the input and convert the output to another DTO instance:
class ExampleEchoInput(DTO): message: str = DTOField('Hello', description='A message') class ExampleEchoOutput(DTO): echo: str input = ExampleEchoInput(message='Hello world') output = ExampleEchoOutput(**self.context.models.example.model(input)) output.echo # will equal 'Hello world from block: 14661701'
You can run a model at a different block number by using the
context.models(block_number=12345) syntax, for example:
# Runs the model with a context of block number 12345 result = self.context.models(block_number=12345).example.model(message='Hello world')
Alternatively you can run a model by slug string using the
def run_model(name: str, input: Union[dict, DTO] = EmptyInput(), return_type: Union[Type[dict], Type[DTO], None], block_number: Union[int, None] = None, version: Union[str, None] = None)
return_type is None or dict, then the method returns the model output as a dict. If it’s a DTO class, the method returns a DTO instance. As above, you can use a dict result with
** to initialize a DTO instance yourself.
# token = Token( ) instance price = Price(**self.context.run_model('price', token)) # has the same effect as: price = self.context.run_model('price', token, return_type=credmark.cmf.types.Price)
context.web3 will return a configured web3 instance with the default block set to the block number of context.
The web3 providers are determined from the environment variables as described in the credmark_dev docs. Currently, during development, model developers will need to use their own alchemy account (or other web3 provider) to access web3 functionality. When a model is deployed, it automatically uses a Credmark web3 provider.
Credmark simplifies the process of getting web3 instances of any contract from any chain. So you don’t need to find and hardcode chain specific attributes and functions within these chains to run your models.
The model context exposes the
context.contracts property which can be used to get contracts by metadata or address. The contracts are instances of the
Contract class which are configured and use the web3 instance at specified block number and specified chain id along with additional data based on
Example code for contact class can be found here.
Currently below parameters as argument are supported to fetched using Contracts:
name: name of the contract
address: address of the contract
deploy_tx_hash: transaction hash at which contract was deployed
protocol: protocol name
product: product name
Contract functions are accessible using the
Tip: the contract object returned from contract class can be used to fetch any specific web3 attributes of the contract and call contract functions. As well it can be used as a DTO (see details below) so it can be returned as part of the output of a model.
Credmark allows access to in-house blockchain ledger data via ledger interface (
context.ledger), so that any model can fetch/use ledger data if required. This is done via
Ledger class which currently supports below functions:
Please refer here for the code of the
context.block_number holds the block number for which a model is running. Models only have access to data at (by default) or before this block number (by instantiating a new context). In other words models cannot see into the future and ledger queries etc. will restrict access to data by this block number.
As a subclass of
BlockNumber class allows the provided block numbers to be treated as integers and hence enables arithmetic operations on block numbers. It also allows you to fetch the corresponding datetime and timestamp properties for the block number. This can be super useful in case we want to run any model iteratively for a certain block-interval or time-interval backwards from the block number provided in the context.
Example code for the block-number class can be found here.
Block number, Timestamp and Python datetime
In blockchain, every block is created with a timestamp (in Unix epoch). In Python there are two types for date, date and datetime, with datetime can be with tzinfo or without. To provide convienent tools to query between the three and resolve the confusion around time, we have a few tools with
block_number.timestamp_datetime: Return the Python datetime with UTC of the block.
block_number.timestamp: Return the Unix epoch of the block.
from_datetime(cls, timestamp: int): Return a
BlockNumberinstance to be less or equal to the input timestamp.
Be cautious when we obtain a timestamp from a Python datetime, we should attach a tzinfo (e.g. timezone.utc) to the datetime. Otherwise, Python take account of the local timezone when converting to a timestamp. See the model
BlockNumberinstance: Obtain a Python datetime with UTC of the block. The block number should be less or equal to the context block.
from credmark.types import BlockNumber dt = BlockNumber(14234904).timestamp_datetime
More example code for the block-number class can be found in here
The historical utility
HistoricalUtil, available at
context.historical (see code here), allows you to run a model over a series of blocks for any defined range and interval.
Block ranges can be specified by blocks (either a window from current block or a start and end block) with
run_model_historical_blocks() or by time (a window from the current block’s time or start and end time) with
run_model_historical(). Times can be specified different units, i.e. year, month, week, day, hour, minute and second.
See e_11_historical.py on how to use this class.