Page History

...

Motivation

In a data warehouse, ingesting MongoDB data is often a complex problem. Although Hive provides a MongoStorageHandler to map MongoDB data into a Hive table, it is practically unusable due to certain characteristics of MongoDB:
Firstly, MongoDB is a document-oriented database that uses BSON format to store data. Its data model is highly flexible, resembling JSON data structures, allowing nesting and inclusion of different data types, making it more adaptable to application requirements.
Secondly, unlike traditional relational databases, MongoDB does not have a fixed data schema. This means that different documents can have different fields and structures. This flexibility allows for easy adjustments and expansion of the data model during the development process, without the need for complex migration procedures.
However, it is precisely these advantages that pose significant challenges during data synchronization. In practice, due to data synchronization limitations, using MongoDB requires sacrificing some flexibility, necessitating that each document be structured. Additionally, upstream business entities may add fields due to business requirements, but downstream systems cannot be aware of these changes, leading to data synchronization issues. Although there are some industry solutions, such as MongoDB CDC (Change Data Capture), synchronizing unstructured JSON data does not work effectively.
Typically, we treat the data in ChangeStream format obtained from MongoDB CDC as a whole and write it into a field in Hive tables. We utilize Hive's Merge functionality to perform updates based on primary keys (_id) and operation timestamps. For business usage, we parse the synchronized JSON data from MongoDB into a JSON table using Hive-JSON-Serde and perform relevant analysis based on that table.
However, while this approach partially addresses some data synchronization issues, it still suffers from high storage costs, long synchronization links, and inconvenience in data usage. Therefore, to resolve the problem of ingesting MongoDB data into the data lake or data warehouse, I suggest developing a SyncAction based on MongoDB, which can assist users in completing the process of ingesting MongoDB data into the data lake or data warehouse.

...