CIM Ingestion Process

Introduction

This guide explains how to ingest CIM (Common Information Model) XML files into OpenDSO using the topology-genesis service (also known as CIMEX). This process is the same regardless of how your CIM file was created—whether exported from utility planning systems, converted from other formats, or hand-crafted for testing.

Prerequisites

Before ingesting CIM files into OpenDSO, ensure you have:

A Valid CIM XML File: Your CIM file can come from any source:
- Utility planning systems (CYME, Synergi, Aspen, etc.)
- GIS or asset management systems
- Conversion tools (CIMHub, vendor tools, etc.)
- Internal utility processes or custom scripts
- For OES testing/development: DSS to CIM or Mav.json to CIM
OpenDSO Environment: A running OpenDSO deployment with at least these containers running:
- Topology-genesis service (CIMEX)
- NATS messaging

Ideally you will want the UI Applications running as well. But you can test CIM Ingestion with just these two

Overview of the Ingestion Process

The CIM ingestion process consists of these main steps:

Prepare CIM File: Validate and prepare the CIM XML file
Configure Ingestion: Set up configuration for topology-genesis (optional)
Run CIMEX: Parse the CIM file and extract equipment data
Verify: Confirm successful ingestion and validate the result

How CIMEX Operates

CIMEX has two operating modes:

1. Ingestion Mode (with -c option):

Parses CIM XML file(s)
Stores data in a local database
Publishes update notifications to NATS
Then switches to service mode

2. Service Mode (without -c option):

Serves data from existing database
Responds to NATS requests for regions, feeders, and devices
No file parsing—works with previously ingested data

Data Flow

CIM XML File → CIMEX Parser → Local Database → NATS Service → Other OpenDSO Services

Quick Start Example

Here's a complete example of ingesting a CIM file using Docker:

# 1. Create directories for your CIM files
mkdir -p resources output

# 2. Place your CIM file in the resources directory
cp my_circuit.xml resources/

# 3. Run CIMEX to ingest the file
docker run -d \
  --name topology_genesis \
  --volume $PWD/resources:/home/resources \
  --volume $PWD/output:/home/output \
  -e PLATFORM_NODE_IP=192.168.1.100 \
  -e PLATFORM_NATS_PORT=4222 \
  oesinc/topology-genesis:latest \
  -c ./resources/my_circuit.xml

# 4. Check the logs
docker logs -f topology_genesis

# 5. Query the ingested data via NATS
nats req opendso.backoffice.cimex.request.regions ''

Step-by-Step Ingestion Process

Step 1: Validate Your CIM File

Before ingestion, validate that your CIM file is complete and well-formed.

Check XML Structure

Ensure the file is valid XML. You can do a simple XML lint check via the xmllint tool.

Example:

# Use xmllint to validate XML syntax
xmllint --noout /path/to/your/circuit.xml

# If valid, no output is shown
# If invalid, errors will be displayed

Verify CIM Content

Check that the CIM file contains the expected elements.

Here are some useful grep commands for verifying the file, and also useful data to have on hand when checking the finalized ingested OpenDSO CIM data.

# Check for Feeder definition
grep -i "cim:Feeder" circuit.xml

# Count equipment types
grep -c "cim:Breaker" circuit.xml
grep -c "cim:ACLineSegment" circuit.xml
grep -c "cim:PowerTransformer" circuit.xml

Step 2: Prepare Equipment Configuration File (Optional)

The equipment configuration file tells CIMEX which equipment types to parse and how to handle connectivity traversal. This file is optional—CIMEX includes default settings.

Example Configuration (cimex.config)

{
  "Equipment": [
    "Breaker",
    "Disconnector",
    "EnergyConsumer",
    "EnergySource",
    "Fuse",
    "GeneratingUnit",
    "Jumper",
    "LinearShuntCompensator",
    "LineSegment",
    "LoadBreakSwitch",
    "PowerElectronicsConnection",
    "PowerTransformer",
    "Recloser",
    "Transformer",
    "TransformerTank",
    "VoltageRegulator"
  ],
  "DontTraverse": ["EnergyConsumer", "EnergySource"],
  "ExcludedEquipment": [
    "PowerTransformer",
    "EnergyConsumer",
    "TransformerTank",
    "Fuse",
    "Disconnector",
    "Jumper",
    "EnergySource"
  ],
  "IgnoredEquipment": [
    "PowerTransformer",
    "TransformerTank",
    "EnergyConsumer",
    "Disconnector",
    "Jumper",
    "Fuse",
    "EnergySource"
  ],
  "NoSiblings": ["LoadBreakSwitch"]
}

Configuration Parameters:

Equipment: List of equipment types to parse from the CIM file
DontTraverse: Equipment types that should not be traversed for connectivity
ExcludedEquipment: Equipment that doesn't define a connectivity stop point
IgnoredEquipment: Equipment that won't appear in region/feeder views
NoSiblings: Equipment types that shouldn't have sibling relationships tracked

Step 3: Run Topology-Genesis (CIMEX)

Running as a Docker Container

The easiest way to run CIMEX is using the pre-built Docker image from Docker Hub.

Requirements:

Docker installed on your system
CIM XML file ready for ingestion
NATS server accessible from the container
Two directories: resources/ for input files and output/ for database storage

Basic Docker run:

# Create directories if they don't exist
mkdir -p resources output

# Place your CIM file in resources directory
cp circuit.xml resources/

# Run CIMEX container
docker run -d \
  --name topology_genesis \
  --volume $PWD/resources:/home/resources \
  --volume $PWD/output:/home/output \
  -e PLATFORM_NODE_IP=<your-nats-server-ip> \
  -e PLATFORM_NATS_PORT=4222 \
  oesinc/topology-genesis:latest \
  -c ./resources/circuit.xml

With custom equipment configuration:

# Place your config file in resources
cp cimex.config resources/

docker run -d \
  --name topology_genesis \
  --volume $PWD/resources:/home/resources \
  --volume $PWD/output:/home/output \
  -e PLATFORM_NODE_IP=<your-nats-server-ip> \
  -e PLATFORM_NATS_PORT=4222 \
  oesinc/topology-genesis:latest \
  -c ./resources/circuit.xml -e ./resources/cimex.config

Environment Variables:

Configure NATS connection using these environment variables:

PLATFORM_NODE_IP or PLATFORM_NATS_SERVICE: NATS server IP address or hostname
PLATFORM_NATS_PORT: NATS server port (default: 4222)

Docker Image versions:

In OpenDSO Deployments, you will find a hash-tag for the docker image. Something like: oesinc/topology-genesis:c53cf381.

c53cf381 (as just an example) is a tagged release of the topology-genesis image. Using tags lock in the current version of the codebase, creating a repeatable deployment.
You can also use the latest tag for the most recent releases, but the image may change between deployments. This is considered a stable release, but you may encounter unexpected changes in behavior if you use latest.
Check with the OpenDSO Support for team for any newer releases.

Running in Docker Compose/OpenDSO Deployment

If you're using OpenDSO's full deployment, CIMEX is typically already running as part of the platform. In this case:

Upload your CIM file to the shared storage volume used by the topology-genesis service
Trigger ingestion by restarting the service with the new file.
Monitor progress through NATS monitoring tools and reviewing the container logs.

Command-Line Options Reference

When running the CIMEX container, you can pass these options after the image name:

-c <file>: CIM XML file(s) to parse (required for ingestion, can specify multiple)
-e <config>: Equipment configuration file (JSON format, optional)
-s <file>: Serialize CIM traversal for debugging purposes
--db <file>: Database file path (default: cim.db in working directory)
--drop: Drop existing database and start fresh
-h: Display help message

Examples:

# Ingest single file with default settings
-c ./resources/circuit.xml

# Ingest with custom equipment config
-c ./resources/circuit.xml -e ./resources/cimex.config

# Ingest multiple files at once
-c ./resources/feeder1.xml -c ./resources/feeder2.xml -c ./resources/feeder3.xml

# Drop existing data and re-ingest
-c ./resources/circuit.xml --drop

# Custom database location
-c ./resources/circuit.xml --db /home/output/custom.db

Note: If you run CIMEX without the -c option, it will operate in service mode only, serving data from the existing database via NATS.

Step 4: Monitor the Ingestion Process

Watch CIMEX Logs

Monitor the CIMEX logs to track progress:

# If using Docker
docker logs -f topology-genesis

# Look for key messages:
# - "Parsing CIM file..."
# - "Found X equipment items"
# - "Processing topology..."
# - "Saving to database..."
# - "Ingestion complete"

Check for Errors

Common log messages and their meanings:

"Invalid XML structure": CIM file has XML syntax errors
"Missing mRID for equipment": Equipment lacks unique identifier
"Connectivity error": Equipment connections are incomplete or invalid
Database errors: Cannot access or write to the database file
Duplicate data: Data already exists (use --drop flag to replace)

Step 5: Verify Successful Ingestion

After ingestion completes, verify that the data was stored correctly.

Query via NATS

Request feeder data through the OpenDSO Data Service:

# Request list of regions (feeders)
nats req opendso.backoffice.cimex.request.regions ''

# Request specific region/feeder
nats req opendso.backoffice.cimex.request.region.f9cfd5e4-949d-487c-9c39-4400ec68a24f ''

# Request specific device
nats req opendso.backoffice.cimex.request.device.<device-mrid> ''

Check OpenDSO UI

If you have the OpenDSO web UI available:

Navigate to the One-Line View
Select your feeder from the dropdown
Verify that equipment appears on the diagram
Check that connections and topology are correct
Click on equipment to view details

Supported Equipment Types

Topology-genesis ingests the following CIM equipment types:

CIM Type	Description
cim:Breaker	Circuit breakers
cim:Recloser	Automatic reclosers
cim:Disconnector	Disconnectors and isolators
cim:LoadBreakSwitch	Load break switches
cim:Fuse	Fuses
cim:Jumper	Jumpers and short connections
cim:PowerTransformer	Power transformers
cim:TransformerTank	Transformer tanks
cim:ACLineSegment	Distribution line segments
cim:EnergyConsumer	Loads and energy consumers
cim:EnergySource	Substation sources
cim:GeneratingUnit	Generating units
cim:LinearShuntCompensator	Capacitor banks and shunt compensators
cim:PowerElectronicsConnection	Power electronics connections
cim:PowerElectronicsConnection + PhotovoltaicUnit	Solar PV systems
cim:PowerElectronicsConnection + BatteryUnit	Battery energy storage
cim:PowerElectronicsConnection + EvChargingUnit	EV charging stations

Special Equipment

Voltage Regulators: Identified as PowerTransformers with TapChangerControl and RatioTapChanger

Controllable Loads: Special load types that can be controlled for demand response

Common Issues and Troubleshooting

Issue: CIM File Not Found

Symptoms: CIMEX logs show "Cannot find CIM file"

Solution:

Verify file path in configuration
Check file permissions
Ensure file is mounted in Docker container (if using Docker)

Issue: Duplicate Data in Database

Symptoms: Errors about existing data or duplicate entries

Solution:

Use --drop flag to clear the existing database and re-ingest
Or manually delete the database file (default: cim.db) before re-running

Issue: Missing Equipment After Ingestion

Symptoms: Equipment count is less than expected

Solution:

Check CIMEX logs for parsing warnings
Verify equipment has proper CIM type
Ensure equipment has valid mRID
Check that equipment is supported by CIMEX
Validate connectivity (disconnected equipment may be skipped)

Issue: Incorrect Topology

Symptoms: Equipment connections wrong in UI

Solution:

Verify CIM Terminal and ConnectivityNode definitions
Check that buses are properly defined
Ensure phases are consistent across connections
Review CIMEX logs for parsing of connectivity

Issue: Missing Equipment Properties

Symptoms: Equipment imported but properties missing

Solution:

Check that CIM file includes property profiles
Verify CIM property names match expected fields
Review CIMEX logs for parsing warnings
Check for unsupported CIM extensions

Issue: Database File Permissions

Symptoms: "Cannot access database" or permission errors

Solution:

Verify write permissions for the database file location
In Docker: Ensure volumes are properly mounted with write access
Check that the output directory exists and is writable
Default database location is the current working directory (cim.db)

Best Practices

1. Always Validate Before Ingestion

Run XML validation
Check for duplicate mRIDs
Verify equipment counts
Review connectivity structure

2. Use Version Control

Keep CIM files in version control
Document changes between versions

3. Test in Development First

Ingest in development environment first
Validate results thoroughly
Only promote to production after validation

4. Monitor Database Size

The database file (cim.db) grows with each ingested feeder
For production deployments, ensure adequate disk space
Use --drop flag when replacing data to avoid accumulation
Consider backing up the database file for recovery purposes

Re-Ingesting Updated CIM Files

If you need to update a circuit model, use the --drop flag to completely replace the database:

# Stop and remove existing container
docker stop topology_genesis
docker rm topology_genesis

# Re-run with updated CIM file and --drop flag
docker run -d \
  --name topology_genesis \
  --volume $PWD/resources:/home/resources \
  --volume $PWD/output:/home/output \
  -e PLATFORM_NODE_IP=<your-nats-server-ip> \
  -e PLATFORM_NATS_PORT=4222 \
  oesinc/topology-genesis:latest \
  -c ./resources/updated_circuit.xml -e ./resources/cimex.config --drop

Alternative: Delete database file manually

# Stop container
docker stop topology_genesis
docker rm topology_genesis

# Delete the database file
rm output/cim.db

# Re-run without --drop flag
docker run -d \
  --name topology_genesis \
  --volume $PWD/resources:/home/resources \
  --volume $PWD/output:/home/output \
  -e PLATFORM_NODE_IP=<your-nats-server-ip> \
  -e PLATFORM_NATS_PORT=4222 \
  oesinc/topology-genesis:latest \
  -c ./resources/updated_circuit.xml -e ./resources/cimex.config

Warning: The --drop flag deletes the entire database, removing all previously ingested feeders and regions.

Performance Considerations

Large CIM Files

For large circuits with thousands of devices:

Memory: Ensure adequate RAM for CIMEX (4GB+ recommended)
Time: Ingestion may take several hours

Multiple Feeders

When ingesting multiple feeders, you can process them in a single run:

docker run -d \
  --name topology_genesis \
  --volume $PWD/resources:/home/resources \
  --volume $PWD/output:/home/output \
  -e PLATFORM_NODE_IP=<your-nats-server-ip> \
  -e PLATFORM_NATS_PORT=4222 \
  oesinc/topology-genesis:latest \
  -c ./resources/feeder1.xml -c ./resources/feeder2.xml -c ./resources/feeder3.xml

All feeders will be stored in the same database and accessible via NATS requests.

Integration with Other OpenDSO Services

After ingestion, the circuit data will be available to:

Topology-Nodes: Real-time topology tracking
GMS (Grid Management System): One-line visualization
DER Dispatch: Managing distributed resources
Historian: Time-series data storage
Event Service: Event processing and monitoring

All services access the circuit model through Topology Genesis NATS endpoints.

References

OpenDSO Documentation

CIM Ingestion Overview - Understanding CIM and why OpenDSO uses it
Topology-Genesis Service Documentation - Technical details of the CIMEX service
CIMEX Data Structures - CIMEX DTOs and data structures

For OES Testing and Development

DSS to CIM Process - Creating test circuits from OpenDSS models
Mav.json to CIM Process - OES's simplified format for lab circuits

Introduction​

Prerequisites​

Overview of the Ingestion Process​

How CIMEX Operates​

Data Flow​

Quick Start Example​

Step-by-Step Ingestion Process​

Step 1: Validate Your CIM File​

Check XML Structure​

Verify CIM Content​

Step 2: Prepare Equipment Configuration File (Optional)​

Example Configuration (cimex.config)​

Step 3: Run Topology-Genesis (CIMEX)​

Running as a Docker Container​

Running in Docker Compose/OpenDSO Deployment​

Command-Line Options Reference​

Step 4: Monitor the Ingestion Process​

Watch CIMEX Logs​

Check for Errors​

Step 5: Verify Successful Ingestion​

Query via NATS​

Check OpenDSO UI​

Supported Equipment Types​

Special Equipment​

Common Issues and Troubleshooting​

Issue: CIM File Not Found​

Issue: Duplicate Data in Database​

Issue: Missing Equipment After Ingestion​

Issue: Incorrect Topology​

Issue: Missing Equipment Properties​

Issue: Database File Permissions​

Best Practices​

1. Always Validate Before Ingestion​

2. Use Version Control​

3. Test in Development First​

4. Monitor Database Size​

Re-Ingesting Updated CIM Files​

Performance Considerations​

Large CIM Files​

Multiple Feeders​

Integration with Other OpenDSO Services​

References​

OpenDSO Documentation​

For OES Testing and Development​

Introduction

Prerequisites

Overview of the Ingestion Process

How CIMEX Operates

Data Flow

Quick Start Example

Step-by-Step Ingestion Process

Step 1: Validate Your CIM File

Check XML Structure

Verify CIM Content

Step 2: Prepare Equipment Configuration File (Optional)

Example Configuration (cimex.config)

Step 3: Run Topology-Genesis (CIMEX)

Running as a Docker Container

Running in Docker Compose/OpenDSO Deployment

Command-Line Options Reference

Step 4: Monitor the Ingestion Process

Watch CIMEX Logs

Check for Errors

Step 5: Verify Successful Ingestion

Query via NATS

Check OpenDSO UI

Supported Equipment Types

Special Equipment

Common Issues and Troubleshooting

Issue: CIM File Not Found

Issue: Duplicate Data in Database

Issue: Missing Equipment After Ingestion

Issue: Incorrect Topology

Issue: Missing Equipment Properties

Issue: Database File Permissions

Best Practices

1. Always Validate Before Ingestion

2. Use Version Control

3. Test in Development First

4. Monitor Database Size

Re-Ingesting Updated CIM Files

Performance Considerations

Large CIM Files

Multiple Feeders

Integration with Other OpenDSO Services

References

OpenDSO Documentation

For OES Testing and Development