Background

BanyanDB is the native storage engine for Apache SkyWalking, designed specifically for observability data (Traces, Metrics, and Logs). As BanyanDB matures into a production-ready storage backend, data portability becomes critical. Users need the ability to move datasets between environments (e.g., from production to staging for debugging) or export data for external analysis in tools like Python/Pandas, Spark, or specialized AI training pipelines.

Currently, BanyanDB supports disaster recovery backups and simple CSV dumps for specific models. This project aims to build a high-performance, comprehensive Export/Import Utility that supports multiple formats and ensures data integrity.

Tasks

• Multi-Format Support: Implement export/import functionality for:

• • Native Binary: High-performance format for BanyanDB-to-BanyanDB migration.

• • Plain Text/Standard: Support for Parquet (optimized for metrics/measures) and JSON/CSV (for human readability).

• Batch & Stream Processing: Ensure the tool can handle massive datasets by implementing chunked data reading and writing to avoid memory bottlenecks.

• Schema Evolution Handling: Implement logic to handle cases where the schema in the exported file differs slightly from the target server's schema.

• Integration with bydbctl: Expose these capabilities through a user-friendly CLI command suite (e.g., bydbctl data export --group=user_logs --format=parquet).

Requirements

• Strong knowledge of Go and concurrency patterns.

• Experience with data serialization formats (Protobuf, Parquet, Apache Arrow).

• Familiarity with gRPC-based API communication.

Background

BanyanDB is the native storage engine for Apache SkyWalking, designed specifically for observability data (Traces, Metrics, and Logs). It utilizes its own query language, BydbQL, which is SQL-like but optimized for time-series and observability schemas. While BydbQL is powerful, non-expert users or SREs in high-pressure situations may find it difficult to construct complex queries for specific traces or aggregated metrics.

The goal of this project is to build an Intelligent Query Agent that leverages Large Language Models (LLMs) to translate Natural Language (NL) into valid BydbQL.

Tasks

• Schema-Aware Prompting: Develop a mechanism to extract BanyanDB metadata (Groups, Streams, Measures, Tag Families) and feed it into the LLM context (RAG - Retrieval-Augmented Generation).

• N2SQL Implementation: Adapt state-of-the-art "Natural Language to SQL" (NL2SQL) techniques to the specific syntax and constraints of BydbQL.

• Verification Loop: Integrate the agent with the existing BydbQL parser to validate generated queries before execution.

• CLI/UI Integration: Implement a "chat" interface or an --ask flag in bydbctl (the BanyanDB CLI tool) to allow users to query data via plain English (e.g., "Show me the top 5 slowest services in the last hour").

Requirements

• Proficiency in Go (BanyanDB's primary language).

• Experience with LLM APIs (OpenAI, Gemini, or local models via Ollama) and orchestration frameworks (LangChain, LangGraph).

• Understanding of Compiler Front-ends (Lexing, Parsing, AST).