Storage

JuliaOS utilizes different storage mechanisms for various types of data, including configuration, runtime state, persistent records, and decentralized options.

Overview

JuliaOS implements a comprehensive storage system with multiple layers to meet different persistence, performance, and decentralization requirements:

1. In-Memory Runtime State: Fast access for active agents/swarms, but volatile

2. SQLite Database: Persistent local storage for configurations, agent/swarm definitions, and transaction history

3. Arweave: Decentralized, permanent storage for archiving or sharing data

4. IPFS: Distributed file storage for content-addressable data

5. Configuration Files: Static backend settings and mappings

6. Log Files: Operational logs for debugging and monitoring

Local Storage: SQLite (Storage.jl)

• Location: ~/.juliaos/juliaos.sqlite

• Purpose: Primary persistent storage for structured data

• Schema Includes:

  • agents: Agent definitions, configurations, and states

  • swarms: Swarm definitions, algorithms, and configurations

  • swarm_agents: Mapping between swarms and their member agents

  • transactions: Blockchain transaction history and status

  • api_keys: Securely stored API keys for external services

  • settings: System-wide and user preferences

  • documents: Text documents for LangChain and RAG applications

  • vectors: Vector embeddings for semantic search

  • tasks: Agent and swarm task history and results

  • memory: Agent memory storage

• Key Features:

  • CRUD operations for all major entities

  • JSON handling for complex configuration fields

  • Transaction tracking and history

  • Settings management

  • Document and vector storage for LangChain integration

  • Efficient querying and indexing

Decentralized Storage: Arweave (ArweaveStorage.jl)

• Purpose: Permanent, decentralized storage for long-term data preservation

• Implementation: Fully implemented in ArweaveStorage.jl with integration through Storage.jl

• Key Features:

  • Store and retrieve agent/swarm data on the Arweave network

  • Upload and download arbitrary files

  • Permanent storage with one-time payment

  • Content addressing for data integrity

  • Immutable storage for audit trails and provenance

• Configuration: Set up via Storage.configure_arweave(gateway_url, wallet_path)

• Use Cases:

  • Archiving important agent configurations and results

  • Storing training data and model weights

  • Creating permanent records of transactions and decisions

  • Sharing agent configurations and swarm algorithms publicly

Distributed Storage: IPFS (IPFSStorage.jl)

• Purpose: Content-addressable, distributed file storage

• Implementation: Implemented in IPFSStorage.jl with integration through Storage.jl

• Key Features:

  • Store and retrieve data using content-based addressing

  • Distributed storage across the IPFS network

  • Efficient for larger files and datasets

  • Content verification through cryptographic hashing

• Configuration: Set up via Storage.configure_ipfs(gateway_url, api_key)

• Use Cases:

  • Storing larger datasets for agent training

  • Sharing data between agents and swarms

  • Distributing agent configurations and algorithms

  • Storing intermediate results from swarm optimizations

Document/Vector Storage (DocumentStorage.jl)

• Purpose: Storage for text documents and vector embeddings to support LangChain applications

• Implementation: Fully implemented in DocumentStorage.jl with SQLite backend

• Key Features:

  • Add, search, and delete documents

  • Store and retrieve vector embeddings

  • Support for semantic search using vector similarity

  • Integration with LangChain for RAG applications

  • Efficient indexing for fast retrieval

• Use Cases:

  • Building knowledge bases for agents

  • Implementing Retrieval-Augmented Generation (RAG)

  • Creating semantic search capabilities

  • Storing agent-generated content with metadata

Memory Storage (MemoryStorage.jl)

• Purpose: Persistent memory for agents to store and retrieve information

• Implementation: Implemented in MemoryStorage.jl with SQLite backend

• Key Features:

  • Key-value storage for agent memory

  • Support for structured and unstructured data

  • Efficient retrieval and updating

  • Memory persistence across agent restarts

• Use Cases:

  • Storing agent state and learned information

  • Maintaining context between agent tasks

  • Implementing agent learning and adaptation

  • Sharing information between agents

Configuration Files (/julia/config/)

• Purpose: Store static configuration needed at startup

• Examples:

  • config.toml: Server settings, ports, and global configurations

  • tokens.json: Token address mappings for different chains

  • dex.json: DEX configurations and contract addresses

  • chains.json: Blockchain network configurations

Log Files (/julia/logs/)

• Purpose: Record operational events, errors, and debug information

• Files: server.log, server_run.log, agents.log, swarms.log, etc.

• Features: Configurable log levels, rotation, and formatting

Data Flow Summary

1. Initialization: Configurations are loaded from SQLite into memory at startup/activation

2. Runtime: Changes happen in memory for performance

3. Persistence: Updates are saved back to SQLite for local persistence

4. Decentralization: Critical data can be stored on Arweave or IPFS for permanence or distribution

5. Transactions: Blockchain transactions are recorded in SQLite for history and tracking

6. Documents/Vectors: Added to SQLite when needed for LangChain and RAG applications

Storage Selection Guidelines

• SQLite: Use for most agent and swarm data, configurations, and transaction history

• Arweave: Use for permanent storage of critical data and public sharing with permanence

• IPFS: Use for larger datasets, content-addressable storage, and distributed access

• In-Memory: Use for high-performance, temporary data during agent/swarm execution

See Data Storage Architecture for more technical details on implementation.